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Executive  Summary 


Organizations  throughout  the  world  have  successfully  invested  in  CMMI-based  process 
improvement.  This  report  presents  quantitative  evidence  from  35  organizations,  several  of 
which  are  larger  enterprises  with  more  than  one  constituent  organization.  The  process 
improvement  efforts  in  these  organizations  include  both  small  and  large  organizational  units 
that  do  business  in  a  variety  of  sectors  and  domains.  They  apply  CMMI  model  practices  to 
software  engineering,  systems  engineering,  and  other  engineering  disciplines.  While  most  of 
the  results  come  from  higher  maturity  organizations,  notable  improvements  also  have  been 
achieved  by  lower  maturity  organizations.  All  of  the  organizations  in  the  report  explicitly 
attribute  their  achievements  to  guidance  provided  by  CMMI. 

Performance  results  are  categorized  and  summarized  by  cost,  schedule,  productivity,  quality, 
customer  satisfaction,  and  return  on  investment  (ROI).  These  categories  include  a  wide 
variety  of  measures,  each  selected  by  the  participating  organizations  to  demonstrate 
improvements  in  areas  of  importance  to  their  particular  business  goals  and  objectives.  While 
the  specific  measures  and  results  differ  among  organizations,  the  overall  results  provide 
ample  proof  of  concept  about  the  potential  of  CMMI-based  process  improvement. 

Median  percent  improvements  over  time  in  the  first  five  categories  and  the  median 
improvement  in  ROI  are  summarized  in  Table  1 .  Further  discussion  of  these  results  is 
followed  in  the  report  by  10  short  case  descriptions.  The  descriptions  provide  more  detail 
about  individual  organizations,  their  process  improvement  journeys,  and  the  measured 
performance  gains  as  a  consequence  of  their  CMMI-based  process  improvement. 


Table  1:  Performance  Improvements  over  Time  by  Category 


Performance  Category 

Median  Improvement 

Cost 

34% 

Schedule 

50% 

Productivity 

61% 

Quality 

48% 

Customer  Satisfaction 

14% 

Return  on  Investment 

4.0  :  1 

The  movement  to  CMMI-based  improvement  has  been  accompanied  by  credible  quantitative 
results  for  all  six  performance  categories.  Many  organizations  have  achieved  improvements 
in  product  quality  and  customer  satisfaction  at  the  same  time  that  they  have  achieved  higher 
productivity,  cost  performance,  and  schedule  performance.  Better  quality  may  not  always  be 
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free,  but  it  can  occur  with  better  project  performance  as  a  result  of  disciplined  process 
improvement. 
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Abstract 


There  is  a  widespread  demand  for  factual  information  about  the  impact  and  benefits  of 
process  improvement  based  on  Capability  Maturity  Model®  Integration  (CMMI)  models. 
Much  has  been  learned  since  the  Software  Engineering  Institute  (SEI)  published  a  special 
report  on  this  topic  over  two  years  ago.  There  now  is  evidence  that  process  improvement 
using  the  CMMI  Product  Suite  can  result  in  improvements  in  schedule  and  cost  performance, 
product  quality,  return  on  investment,  and  other  measures  of  performance  outcome. 

This  technical  report  summarizes  much  of  the  publicly  available  empirical  evidence  about  the 
performance  results  that  can  occur  as  a  consequence  of  CMMI-based  process  improvement. 
In  addition,  the  report  contains  a  series  of  brief  case  descriptions  that  were  created  with 
collaboration  from  representatives  from  10  organizations  that  have  achieved  notable 
quantitative  performance  results  through  their  CMMI-based  improvement  efforts. 

The  report  is  meant  for  members  of  engineering  process  groups,  middle  and  first-line 
management,  and  other  potential  process  improvement  participants  who  wish  to  learn  more 
about  how  CMMI  can  contribute  to  measurable  improvements.  It  also  may  be  useful  for 
executives  and  senior  managers  who  are  faced  with  decisions  about  the  allocation  of  scarce 
resources  for  improvement  efforts. 
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1  Introduction 


Many  organizations  throughout  the  world  have  invested  in  CMMI-based  process 
improvement.  Many  of  these  organizations  have  achieved  and  sometimes  surpassed  their 
improvement  goals.  The  achievement  of  process  capability  and  maturity  level  goals  is  an 
important  benchmark  of  success;  however,  it  is  not  enough.  Organizations  undertake  process 
improvement  to  achieve  business-related  performance  goals. 

There  now  is  ample  evidence  that  process  improvement  using  the  CMMI  Product  Suite  can 
result  in  marked  improvements  in  schedule  and  cost  performance,  product  quality,  return  on 
investment,  and  other  measures  of  performance  outcome.  This  document  summarizes  much 
of  the  publicly  available  empirical  evidence  about  the  performance  results  that  can  occur  as  a 
consequence  of  CMMI-based  process  improvement. 

In  some  instances,  the  movement  to  CMMI-based  improvement  has  been  accompanied  by 
marked  jumps  in  performance.  In  other  instances,  the  improvement  in  performance  has 
continued  in  patterns  comparable  to  what  the  organizations  experienced  with  the  CMMI 
source  models  and  other  improvement  frameworks.  In  all  of  the  instances  reported  here,  the 
organizations  have  attributed  their  successes  to  the  guidance  for  improvement  that  is  included 
in  CMMI. 

The  results  presented  in  this  report  come  from  a  variety  of  sectors  including  the 
telecommunications,  financial,  manufacturing,  and  defense  industries.  Many  of  the  results 
come  from  higher  maturity  organizations;  however,  there  also  are  some  impressive  results 
from  lower  maturity  organizations.  Similarly,  most  results  come  from  larger  organizational 
settings,  but  a  few  are  from  smaller  settings. 

At  the  time  of  this  document’s  publication,  almost  all  of  the  evidence  presented  here  was 
available  on  the  CMMI  Performance  Results  Web  site  at 

http://www.sei.cmu.edu/cmmi/results.html.  The  Web  site  also  included  much  more 
information  than  could  be  discussed  in  detail  here.  Other  information  was  provided  initially 
for  use  in  this  document. 

1.1  CMMI-Based  Improvement 

Perspectives  vary  on  what  constitutes  CMMI-based  improvement.  CMMI  models  contain 
content  that  is  presented  differently  or  did  not  exist  in  its  source  documents,  and  CMMI 
provides  guidance  about  integrating  improvement  efforts  across  multiple  disciplines. 
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Organizations  craft  their  process  assets  and  improvement  efforts  in  a  variety  of  ways.  Many 
have  relied  heavily  over  the  years  both  on  CMMI  and  its  source  models.  Others  have 
incorporated  other  improvement  frameworks,  many  of  which  have  a  shared  heritage  with 
CMMI  in  Total  Quality  Management  (TQM). 

Regardless  of  the  other  process  improvement  frameworks  which  organizations  may  have 
used  in  addition  to  CMMI,  all  of  the  evidence  reported  here  comes  from  organizations  that 
explicitly  attribute  their  performance  results  to  guidance  provided  by  CMMI.  In  a  few 
instances,  the  evidence  comes  from  organizations  that  already  had  adopted  practices  that  later 
became  part  of  CMMI. 

1 .2  Intended  Audience  and  Structure  of  the  Report 

This  technical  report  is  meant  to  be  useful  for  members  of  engineering  process  groups, 
middle  and  first-line  management,  and  other  potential  process  improvement  participants  who 
wish  to  learn  more  about  how  CMMI  can  contribute  to  measurable  improvements  in 
organization  and  project  performance  and  product  quality.  It  also  may  be  useful  for 
executives  and  senior  managers  who  are  faced  with  decisions  about  the  allocation  of  scarce 
resources  for  improvement  efforts. 

Section  2  of  this  document  contains  a  brief  discussion  of  the  approach  used  for  demonstrating 
the  impact  on  project  performance  and  product  quality  of  CMMI-based  process 
improvement.  It  is  based  on  a  section  of  the  same  name  in  the  report  titled  Demonstrating  the 
Impact  and  Benefits  of  CMMI:  An  Update  and  Preliminary  Results  [Goldenson  03]. 

Section  3  summarizes  the  results  to  date  and  provides  a  more  detailed  breakdown  of  results 
by  performance  categories:  cost,  schedule,  productivity,  quality,  customer  satisfaction,  and 
return  on  investment.  The  section  also  includes  some  brief  comparisons  of  results  by  maturity 
level,  model,  and  organization  scope. 

Section  4  contains  a  series  of  brief  case  descriptions  that  were  created  with  collaboration 
from  representatives  from  10  organizations  that  have  achieved  notable  quantitative 
performance  results  through  their  CMMI-based  improvement  efforts.  A  summary  and 
conclusions  are  in  Section  5. 
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2  Demonstrating  Impact  on  Performance 


Organizations  differ  in  their  business  goals  and  strategic  objectives  as  well  as  the  products 
and  services  that  they  provide.  They  differ  in  how  they  implement  CMMI-based  process 
improvement  and  in  the  ways  they  measure  their  resulting  progress  and  performance. 

Process  improvement  based  on  CMMI  models  may  be  demonstrated  in  several  ways.  Some 
organizations  have  established  new  processes  or  changed  existing  processes  as  a  result  of 
guidance  found  in  the  CMMI  Product  Suite.  Others  have  broadened  the  organizational  scope 
of  their  improvement  efforts,  through  the  integration  of  systems,  software,  hardware,  and 
related  disciplines.  Process  changes  also  sometimes  predate  the  transition  to  CMMI  models, 
especially  for  those  organizations  that  were  among  the  first  to  have  been  appraised  at  the 
higher  maturity  levels  of  CMMI.  Some  organizations,  especially  those  that  had  used  the 
Electronics  Industry  Alliance  System  Engineering  Capability  Model  (EIA/IS  731)  or  the 
early  Systems  Engineering  CMM®  had  already  established  processes  that  were  subsequently 
included  in  CMMI.  Not  all  of  their  experiences  can  be  attributed  to  CMMI  adoption,  but  their 
results  certainly  show  the  importance  of  the  practices  that  are  articulated  in  CMMI  models. 

Analytic  approaches  also  differ.  Some  organizations  make  comparisons  over  time,  across 
maturity  levels,  or  relative  to  other  major  process  milestones.  Others  focus  on  selected  project 
or  organizational  processes  that  they  have  mapped  to  CMMI  model  practices.  Similarly, 
many  measures  of  performance  can  be  used  to  demonstrate  the  impact  of  CMMI-based 
process  improvement.  Organizations  often  describe  CMMI  benefits  in  different  ways,  and 
they  use  the  measures  that  make  the  most  sense  given  their  particular  business  goals. 

Figure  1  is  a  high-level  depiction  of  the  impact  of  process  improvement.  It  is  used  in  this 
document  for  CMMI-based  improvement;  however,  the  same  depiction  can  be  applied 
elsewhere.  The  costs  of  process  improvement  are  illustrated  in  the  upper  left.  Some  of  these 
may  be  planned  investments  for  process  improvement;  others  may  be  expenses  directly  or 
indirectly  related  to  process  improvement.  Process  capability  and  organizational  maturity  are 
depicted  in  the  center  box.  Organizations  improve  their  processes  in  order  to  realize  other 
benefits,  and  they  use  process  capability  and  organizational  maturity  to  evaluate  and  compare 
their  results.  The  box  on  the  right  illustrates  the  kinds  of  benefits  that  organizations  most 
frequently  seek  to  attain  as  a  result  of  their  process  improvement  efforts.  The  costs  and 
benefits  also  can  be  combined  to  calculate  ROI  or  related  measures,  as  shown  in  the  box  on 
the  bottom  of  the  figure. 
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Figure  1:  High-Level  Model  of  CM  Ml  Impact 

The  performance  measures  are  classified  into  six  broad  categories  for  this  report.  As  shown  in 
the  detail  in  Figure  1,  potential  benefits  of  process  improvement  might  accrue  with  respect  to 
cost,  schedule,  productivity,  quality,  and  customer  satisfaction.  The  sixth  category  is  return  on 
investment  and  related  measures,  as  shown  in  the  bottom  box.  Improvements  in  the  six 
categories  can  contribute  to  additional  business  goals,  for  example,  market  share,  reduced 
time  to  market,  lower  cost  products,  and  higher  quality  products. 

Organizations  typically  seek  to  optimize  some  combination  of  the  five  primary  classes  of 
benefits;  each  of  these  benefits  can  be  refined  to  include  a  number  of  more  specific  measures. 
For  example,  one  organization  may  be  more  interested  in  reducing  the  costs  of  its  products 
and  services  while  another  may  be  concerned  about  having  more  predictable  project  costs  or 
schedules.  Other  refinements  of  the  five  basic  categories  might  include  shorter  cycle  times, 
improved  reliability,  reductions  in  rework  and  concomitant  effort,  enhanced  functionality, 
maintainability,  and  reduced  cost  over  the  life  of  the  product. 

As  shown  in  the  box  at  the  bottom  of  Figure  1,  ROI  and  related  cost-benefit  measures 
constitute  the  sixth  category  of  performance  measures.  Organizations  frequently  are 
concerned  about  ROI  measures,  for  which  they  need  to  track  data  that  shows  related  costs 
and  benefits. 

Other  possible  performance  measures,  such  as  market  share,  business  growth,  profitability, 
and  employee  morale,  per  se  do  not  map  well  to  the  simple  benefits  classification  presented 
in  Figure  1.  Such  measures  are  discussed  in  the  case  descriptions  in  Section  4. 
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3  Performance  Results 


Section  3.1  contains  a  high-level  review  of  the  quantitative  performance  results  that  were 
available  at  the  time  this  document  was  published.  Additional  guidance  for  interpreting  those 
results  is  found  in  Section  3.2.  Section  3.3  follows  with  a  more  detailed  breakdown  by 
performance  category,  including  some  brief  comparisons  by  maturity  level. 

3.1  Summarizing  Performance  Results 

The  quantitative  performance  results  in  Table  2  are  from  a  total  of  35  organizations,  some  of 
which  are  enterprises  with  more  than  one  constituent  organization.  30  of  them  have  results 
that  can  be  expressed  as  change  over  time.  These  results  are  expressed  either  as  percentage 
change  from  an  earlier  baseline  prior  to  the  CMMI-based  process  improvement  or  as  ratios  of 
return  on  investment  (ROI).  The  results  are  summarized  by  the  six  performance  categories 
discussed  in  Section  2  of  this  document:  cost,  schedule,  productivity,  quality,  customer 
satisfaction  and  return  on  investment.  Most  of  the  organizations  have  provided  multiple 
results,  sometimes  several  in  the  same  performance  category. 


Table  2:  CMMI  Performance  Results  Summary 


Performance 

Category 

Median 

Improvement 

Number  of  Data 
Points 

Lowest 

Improvement 

Highest 

Improvement 

Cost 

34% 

29 

3% 

87% 

Schedule 

50% 

22 

2% 

95% 

Productivity 

61% 

20 

11% 

329% 

Quality 

48% 

34 

2% 

132% 

Customer 

Satisfaction 

14% 

7 

-4% 

55% 

Return  on  Investment 

4.0:  1 

22 

1.7  :  1 

27.7  :  1 

Note:  The  performance  results  in  this  table  express  change  over  varying  periods  of  time. 

The  summary  results  in  Table  2  and  the  rest  of  this  document  provide  substantial  evidence 
about  the  performance  results  that  are  possible  by  adherence  to  CMMI-based  processes.  The 
median  results  in  every  category  of  the  table  are  notable,  as  are  both  the  high  and  low  ends  of 
the  distributions. 
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The  results  in  Table  2  come  from  several  sources.  Most  of  these  sources  are  public  reports, 
typically  conference  presentations  and  published  papers  by  individuals  from  CMMI- 
appraised  organizations.  These  sources  can  be  seen  in  Appendix  A.  The  table  also  includes 
the  results  provided  by  the  organizations  with  which  the  authors  have  collaborated  in 
preparing  the  brief  case  descriptions  in  Section  4.  In  almost  all  instances,  the  results  have 
been  corroborated  though  direct  communication  with  members  of  the  participating 
organizations;  three  of  which  have  chosen  to  remain  anonymous. 

A  large  number  of  performance  results  are  not  included  in  the  table  because  they  could  not  be 
expressed  as  change  over  time.  These  results  are  equally  interesting  and  valid  although  not  as 
easily  summarized.  They  have  been  included  in  Section  3.3,  where  the  results  are  discussed 
by  performance  category. 

3.2  Interpreting  the  Results 

As  mentioned  in  Section  2,  the  six  performance  categories  are  quite  broadly  defined.  They 
combine  data  points  from  a  wide  variety  of  cases,  ranging  from  pilot  projects  about  the 
results  associated  with  particular  processes  to  major  organization-wide  improvement 
initiatives  covering  the  full  scope  of  CMMI.  The  CMMI-based  improvements  also 
occasionally  occur  simultaneously  with  other  factors  such  as  increased  reuse,  personnel 
changes,  or  the  introduction  of  other  new  technologies. 

The  specific  measures  used  by  the  various  reporting  organizations  also  differ.  For  example,  as 
shown  in  Table  3,  the  cost  category  includes  reductions  in  cost,  cost  performance,  and  related 
measures  of  discrepancy  between  estimates  and  actual  results.  Similar  differences  for 
schedule,  productivity,  and  ROI  are  shown  in  Table  4,  Table  5,  and  Table  6.  There  are  no 
tables  for  the  quality  and  customer  satisfaction  categories.  Quality  measures  are  almost 
always  expressed  as  defects  although  the  specific  measures  vary  depending  on  the 
information  needs  of  the  reporting  organizations  (e.g.,  phase  of  injection,  phase  of  discovery, 
or  total  density).  Award  fees  and  customer  survey  data  are  the  only  common  measures  of 
customer  satisfaction. 

As  more  evidence  becomes  available,  the  SEI  will  refine  the  existing  categories  and  begin 
classifying  the  respective  data  points  separately  wherever  possible.  The  variety  in  each  of  the 
categories  does  highlight  the  multiplicity  of  techniques  used  by  organizations  to  measure 
their  processes  and  products,  and  the  wide  variety  of  ways  they  use  to  present  the  results 
relevant  to  their  organizations. 
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Table  3:  Variety  in  Measures  of  Cost 


Reductions  in 

Costs  (general  category )/Cost  of  delivery 

Cost  of  quality/Cost  of  poor  quality 

Overhead  rate 

Costs  of  rework/Defect  find  &  fix  costs 

Number/Cost  of  process  staff 

Software  unit  costs 

Variation  in  cost  performance  index  (CPI) 

Savings  in  or  due  to 

Implementing  hardware  processes 

Improved 

Budget  estimation  accuracy 

Average  cost  performance  index  (CPI) 

Table  4:  Variety  in  Measures  of  Schedule 


Reductions  in 

Variation  in  schedule 

Schedule  performance  index  (SPI) 

Number  of  days  late 

Days  variance  from  plan 

Slippage  of  project  delivery 

Improved  or  increased 

Cycle  time 

Average  schedule  performance  index  (SPI) 

Proportion  of  milestones  met 

Estimation  accuracy 

Table  5:  Variety  in  Measures  of  Productivity 


Lines  of  code  per  labor  hour 

Number  of  releases  per  year 

Source  statements  per  month 

Testing  rates 

Function  points  per  full  time  equivalent  staff 

Time  comparisons  by  build 

Software  production  (general  category) 

Table  6:  Variety  in  Measures  of  Return  on  Investment 


Example  benefits  and  costs  avoided 

Examples  of  improvement  investment 

Rework  avoided  due  to  fewer  defects 

Quality  activities 

Improved  productivity 

Automation 

Increased  revenue  due  to  shorter  cycle  time 

Process  improvement  in  general 

While  it  would  be  helpful  to  have  a  common  set  of  measures  for  benchmarking  and 
comparison,  valuable  conclusions  still  can  be  drawn  from  the  results  summarized  here. 
Consistent  results  about  the  same  general  classes  of  phenomena  that  are  based  on  different 
measurements  and  data  analytic  procedures,  such  as  those  found  in  this  document,  are  highly 
unlikely  to  occur  by  chance  alone  [Campbell  66,  Cook  79].  These  results  provide  ample 
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proof  of  concept  about  the  potential  of  CMMI-based  process  improvement.  Of  course,  proof 
of  concept  is  not  the  same  thing  as  a  guarantee  of  success,  and  these  same  results  may  not  be 
repeatable  in  different  organizations. 

Much  of  the  evidence  seen  throughout  the  history  of  model-based  process  improvement  is 
limited  to  positive  results.  Organizations  rarely  are  willing  to  discuss  their  problems  and 
disappointments  publicly.  Some  of  the  source  documents  examined  for  this  report  do  show  a 
failure  to  fully  reach  performance  goals;  however,  some  of  these  organizations  also  report 
progress  toward  meeting  those  goals.  Only  one  data  point  in  Table  2  is  a  negative  number, 
and  it  shows  a  slight  change  in  what  remains  an  excellent  level  of  customer  satisfaction. 

Most  of  the  quantitative  performance  results  summarized  in  this  report  come  from  CMMI 
maturity  level  4  and  5  organizations.  It  takes  time  to  establish  a  measurement  capability, 
determine  what  quantitative  measures  may  be  useful  in  a  particular  organizational  context, 
and  accumulate  data. 

Results  showing  improvements  also  are  available  from  lower  maturity  organizations  (defined 
here  as  maturity  levels  1,  2,  and  3).  The  initial  improvements  experienced  by  these 
organizations  may  appear  to  be  comparably  high;  however,  individual  results  should  not  be 
over  generalized  out  of  context.  Percentages  based  on  initially  low  values  leave  more  room 
for  improvement.  Particular  processes  and  measures  also  differ,  and  the  lower  maturity 
organizations  from  which  quantitative  results  are  available  may  very  well  be  atypical  and 
exemplary.  The  important  thing  to  note  here  simply  is  that  there  is  proof  of  concept  that  lower 
maturity  organizations  also  can  achieve  important  performance  gains  through  CMMI-based 
process  improvement. 

Many  organizations  that  successfully  improve  their  processes  may  achieve  performance 
results  that  are  comparable  to  those  described  in  this  document.  Nevertheless,  more 
quantitative  work  remains  to  be  done  to  investigate  the  circumstances  under  which  the 
necessary  process  capabilities  can  be  improved  and  sustained. 
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3.3  Further  Detail 


This  section  provides  further  detail  about  each  of  the  six  performance  categories.  A  separate 
figure  for  each  category  provides  a  better  sense  of  the  range  of  differences  within  that 
category  than  can  be  seen  in  Table  2.  Each  figure  shows  the  full  distribution  of  the  changes 
over  time  in  its  respective  category.  All  but  one  of  the  figures  display  results  for  both  lower 
and  higher  maturity  organizations.  With  the  addition  of  the  Measurement  and  Analysis 
process  area  at  maturity  level  2  in  CMMI,  more  results  from  lower  maturity  organizations 
may  become  available. 

Remember  also  that  the  performance  categories  used  in  this  report  include  a  wide  variety  of 
measures  because  organizations  use  the  measures  that  make  the  most  sense  for  their 
individual  information  needs  and  business  objectives.  This  can  be  seen  in  the  example 
assertion  statements  and  further  discussion  that  follows  for  each  of  the  six  performance 
categories.  The  assertion  statements  are  summary  descriptions  of  the  measures  and  results 
provided  by  the  organizations  in  this  report.  Source  information  for  all  of  the  assertions  can 
be  found  in  Appendix  A.  Additional  detail  and  more  examples  can  be  found  on  the  CMMI 
Performance  Results  Web  site  at  http://www.sei.cmu.edu/cmmi/results.html. 

3.3.1  Cost 

The  cost  category  covers  instances  where  organizations  report  changes  in  the  cost  of  final  or 
intermediate  work  products,  changes  in  the  cost  of  the  processes  employed  to  produce  the 
products,  and  general  savings  attributed  to  model-based  process  improvement.  It  also 
includes  increased  predictability  of  costs  incurred  and  other  measures  of  variation. 

Figure  2  shows  5  data  points  on  cost  performance  improvements  over  time  from  lower 
maturity  organizations  and  24  data  points  from  organizations  that  have  achieved  maturity 
level  4  or  5.  The  performance  results  from  the  lower  maturity  organizations  sometimes  may 
appear  to  exceed  those  found  in  the  higher  maturity  organizations;  however,  the  magnitude  of 
the  percentage  changes  should  not  be  over-generalized.  Relatively  small  changes  from 
initially  low  baselines  can  appear  to  be  quite  high  when  stated  in  percentage  terms.  The 
individual  processes  also  differ,  as  do  the  measures  of  the  performance  results. 
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Distribution  in  rank  order  from  low  to  high 

♦  Lower  Maturity  Organizations  (Maturity  Levels  1-3) 
a  Higher  Maturity  Organizations  (Maturity  Levels  4  and  5) 


These  data  points  include 

•  pilot  projects  focused  on  particular  processes 

•  major  organization-wide  improvement  initiatives 

•  a  wide  variety  of  measures  based  on  differing  information  needs 


Figure  2:  Changes  Over  Time  in  Cost 

Table  7  includes  selected  assertion  statements  that  describe  the  kinds  of  cost  predictability 
and  performance  gains  that  have  been  achieved  through  CMMI-based  process  improvement. 
The  first  three  table  entries  are  from  lower  maturity  organizations. 

The  statement  in  the  first  row  of  the  table  describes  cost  savings  due  to  improved  hardware 
engineering  practices  at  a  medium-sized  aerospace  company  that  had  implemented  CMMI. 
The  statement  in  the  second  table  row  describes  the  results  from  an  IT  company  that  had  just 
begun  its  use  of  model-based  process  improvement  yet  significantly  reduced  its  cost  of 
quality  in  one  year.  As  seen  later  in  this  technical  report,  this  company’s  improvement  in  cost 
performance  was  accompanied  by  measured  improvement  in  productivity  and  return  on 
investment. 


Table  7;  Summary  Benefits  and  Impact — Cost 


# 

Result 

Organization 

1 

$2.1  Million  in  savings  in  hardware  engineering  processes  in  an  organization 
moving  toward  CMMI  maturity  level  3 

Anonymous  1 

10 
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# 

Result 

Organization 

2 

Used  CMMI  Measurement  and  Analysis  process  area  to  significantly  reduce 
cost  of  quality  in  one  year 

Anonymous  2 

3 

Costs  dropped  52  percent  from  a  baseline  prior  to  SW-CMM  maturity  level  2  as 
the  organization  moved  toward  CMMI  maturity  level  3 

DB  Systems  GmbH 

4 

On-budget  delivery  improved  from  over  90  percent  to  nearly  1 00  percent  as  the 
organization  moved  from  SW-CMM  maturity  level  3  to  CMMI  maturity  Level 

5 

IBM  Australia 

Application 

Management  Services 

5 

Substantially  improved  cost  variance  over  three  causal  analysis  and  resolution 
cycles  in  a  CMMI  maturity  level  5  organization  with  PSPSM-trained  engineers 

Northrop  Grumman  IT, 
Defense  Enterprise 
Solutions 

6 

CMMI  maturity  level  3  site  reduced  its  costs  of  rework  by  42  percent  over 
several  years 

Raytheon  Corp 
Anonymous  site 

7 

Reduced  cost  of  poor  quality  from  over  45  percent  to  under  30  percent  over  a 
three  year  period  as  the  organization  moved  from  SW-CMM  maturity  level  5 
towards  CMMI  maturity  level  5 

Siemens  Information 
Systems  Ltd. 

8 

Estimation  accuracy  improved  by  72  percent  on  average  in  three  technical  areas 
from  1996  through  2004  On  average,  63  percent  of  that  improvement  occurred 
as  the  organization  moved  from  SW-CMM  maturity  level  5  to  CMMI  maturity 
level  5 

Siemens  Information 
Systems  Ltd. 

9 

$95K  in  savings  attained  by  reduced  rework  effort  through  early  detection  of 
memory  leaks  in  a  CMMI  maturity  level  5  organization 

Tata  Consultancy 

Services 

10 

Additional  3  percent  increase  in  both  code  reuse  and  related  material  reuse  as 
the  organization  moved  from  SW-CMM  maturity  level  4  to  CMMI  maturity 
level  5 

The  Boeing  Company 

The  second  anonymous  company  had  begun  to  work  toward  SW-CMM  maturity  level  2 
when  CMMI  was  first  released.  Management  was  reticent  to  adopt  CMMI  right  away; 
however,  the  organization’s  consultant  used  the  CMMI  Measurement  and  Analysis  process 
area  to  base  the  company’s  improvement  effort  on  quantitative  evidence.  The  value  of  doing 
this  was  evident  to  the  company’s  management  by  the  time  of  its  SW-CMM  assessment. 

The  statement  in  row  three  of  Table  7  is  from  DB  Systems  GmbH,  a  major  IT  provider  for  the 
transport  and  travel  industry.  As  seen  in  Figure  3,  20  percent  of  the  drop  in  costs  at  DB 
Systems  GmbH  came  as  the  organization  moved  to  SW-CMM  maturity  level  2  while  an 
additional  32  percent  was  achieved  as  the  organization  approached  CMMI  maturity  level  3. 
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Figure  3:  Cost  Savings  at  DB  Systems  GmbH 

Results  from  Northrop  Grumman  Information  Technology,  Defense  Enterprise  Solutions  also 
are  particularly  compelling.  Expanding  on  the  fifth  statement  in  Table  7,  Figure  4  shows  the 
changes  between  estimated  and  actual  costs  for  the  development  of  a  firm-fixed  price 
inventory  tracking  system  over  a  three  and  a  half  year  period.  The  project  went  through  three 
causal  analysis  and  resolution  cycles  to  improve  its  peer  review  processes.  During  that  time 
the  organization  completed  five  major  builds  of  comparable  difficulty.  Its  process  changes 
were  accompanied  by  a  sizable  improvement  in  cost  performance.  In  build  1,  the  Northrop 
Grumman  IT  project  substantially  underestimated  its  costs.  By  build  6,  the  organization  was 
able  to  complete  the  work  for  substantially  less  cost  than  initially  estimated. 
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Figure  4:  Cost  Savings  at  Northrop  Grumman  IT 

As  seen  later  in  this  technical  report,  the  Northrop  Grumman  IT  project  achieved  similar 
results  for  schedule  performance,  product  quality,  and  return  on  investment.  Comparable 
results  from  other  organizations  are  described  in  more  detail  in  the  case  descriptions  in 
Section  4.  Many  other  examples  of  cost,  schedule,  and  productivity  improvements  in  concert 
with  improved  product  quality,  customer  satisfaction,  and  return  on  investment  can  be  found 
throughout  this  report  and  on  the  CMMI  Performance  Results  Web  site. 

3.3.2  Schedule 

This  category  covers  improvements  in  schedule  predictability  and  reductions  in  the  time 
required  to  do  the  work.  Figure  5  provides  a  better  sense  of  the  full  range  of  differences  in 
schedule  performance  than  can  be  seen  in  Table  2.  As  seen  in  Figure  5,  there  are  six  data 
points  from  lower  maturity  level  organizations  that  reported  improvements  over  time  in 
schedule  performance.  In  fact,  four  of  the  largest  percent  improvements  in  schedule 
performance  come  from  lower  maturity  organizations.  Once  again,  there  are  many  possible 
reasons  why  the  lower  maturity  level  organizations  have  achieved  relatively  high  percentage 
improvements.  Individual  percentage  differences  should  not  be  over-interpreted. 
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Distribution  in  rank  order  from  low  to  high 


♦  Lower  Maturity  Organizations  (Maturity  Levels  1-3) 
a  Higher  Maturity  Organizations  (Maturity  Levels  4  and  5) 


These  data  points  include 

•  pilot  projects  focused  on  particular  processes 

•  major  organization-wide  improvement  initiatives 

•  a  wide  variety  of  measures  based  on  differing  information  needs 


Figure  5:  Changes  Over  Time  in  Schedule 

Table  8  includes  a  few  selected  assertion  statements  that  describe  the  kinds  of  schedule 
predictability  and  performance  gains  that  have  been  achieved  through  CMMI-based  process 
improvement. 


Table  8:  Summary  Benefits  and  Impact — Schedule 


# 

Result 

Organization 

1 

On-time  delivery  remained  well  over  90  percent,  moving  from  97%  to  99%, 
with  a  slight  improvement  as  the  organization  moved  from  SW-CMM  maturity 
level  3  to  CMMI  maturity  level  5 

IBM  Australia 

Application 

Management  Services 

2 

70  to  80  percent  reduction  in  average  slippage  of  project  delivery  dates  as  the 
organization  achieved  CMMI  maturity  level  2 

JP  Morgan  Chase 

3 

Average  days  variance  from  development  plan  reduced  from  approximately  130 
days  to  less  than  20  days  one  year  after  reaching  CMMI  maturity  level  2 

NCR 

4 

Met  every  schedule  milestone  (25  in  a  row)  on  time,  with  high  quality  and 
customer  satisfaction  in  a  CMMI  maturity  level  5  organization 

Northrop  Grumman  IT, 
Defense  Enterprise 
Solutions 

5 

Substantially  improved  schedule  variance  over  three  causal  analysis  and 
resolution  cycles  in  a  CMMI  maturity  level  5  organization  with  PSP-trained 
engineers 

Northrop  Grumman  IT, 
Defense  Enterprise 
Solutions 

14 
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# 

Result 

Organization 

6 

Schedule  variance  improved  from  approximately  25  percent  to  15  percent  as  the 
organization  moved  from  SW-CMM  maturity  level  3  to  CMMI  maturity  level  5 

Reuters 

7 

On-time  deliveries  improved  from  79  percent  to  89  percent  as  the  organization 
moved  from  SW-CMM  maturity  level  3  toward  CMMI  maturity  level  4 

Systematic  Software 
Engineering 

8 

Schedule  variation  decline  by  63  percent  as  the  organization  moved  from  SW- 
CMM  maturity  level  4  to  CMMI  maturity  level  5 

The  Boeing  Company 

Note  for  example,  the  third  statement  in  the  table.  It  is  from  NCR  Self  Service  in  Scotland, 
which  had  been  appraised  at  CMMI  maturity  level  2.  As  seen  in  Figure  6,  this  organization 
had  a  noticeable  reversal  immediately  after  it  began  its  move  to  CMMI.  As  it  worked  toward 
CMMI  maturity  level  3,  this  organization  progressed  from  a  little  over  60  days  average 
variance  from  plan  at  SW-CMM  maturity  level  3  to  a  little  less  than  20  days. 


Figure  6:  Schedule  Improvement  at  NCR  Self  Service,  Scotland 

Figure  7  shows  the  marked  improvement  in  schedule  performance  that  mirrors  the  cost 
performance  improvements  at  Northrop  Grumman  IT  shown  in  Section  3.3.1.  The 
organization  completed  its  first  build  one  percent  ahead  of  schedule  and  the  second  build  one 
percent  behind  schedule.  It  completed  the  fifth  build  in  a  little  less  than  half  the  time 
originally  planned.  As  seen  in  Table  8,  the  same  organization  met  every  scheduled  milestone 
on  time  with  high  quality  and  customer  satisfaction. 
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Figure  7:  Schedule  Improvement  at  Northrop  Grumman  IT 

3.3.3  Productivity 

This  category  includes  various  measures  based  on  the  amount  of  work  accomplished  in  a 
given  period  of  time.  Figure  8  shows  the  full  range  of  differences  in  productivity,  six  of 
which  come  from  lower  maturity  level  organizations.  Most  of  the  productivity  gains  fall  in 
the  range  between  10  and  100  percent  improvement  (i.e.,  the  best  among  them  have  doubled 
their  initial  productivity  as  a  result  of  their  CMMI-based  process  improvement  efforts).  The 
most  extreme  outlier  is  from  improved  defect  detection  practices  in  an  organization  that 
already  was  progressing  toward  CMMI  maturity  level  4.  The  next  highest  data  point  is  from  a 
single  project  in  a  higher  maturity  level  organization.  The  third  highest  data  point  is  from  a 
higher  maturity  level  organization  that  included  value  added  by  new  business  in  its  measure 
of  productivity  gain.  These  three  measures  were  useful  in  their  own  organizations’  business 
contexts;  however,  these  results  do  highlight  the  shortcomings  of  using  dissimilar  measures 
when  comparing  organizations. 
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Table  9  includes  selected  assertion  statements  that  describe  the  kinds  of  productivity  gains 
that  have  been  achieved  through  CMMI-based  process  improvement.  Notice  in  particular  the 
first  table  entry.  It  shows  a  marked  productivity  increase  in  the  same  anonymous  organization 
that  also  had  lowered  its  cost  of  quality  during  its  first  year  of  CMM-based  improvement  by 
using  the  Measurement  and  Analysis  process  area  to  structure  its  improvement  effort  (see  the 
discussion  in  Section  3.3.1).  As  with  cost  and  schedule  performance,  most  of  the  reported 
productivity  gains  are  reported  by  higher  maturity  organizations. 
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These  data  points  include 

•  pilot  projects  focused  on  particular  processes 

•  major  organization-wide  improvement  initiatives 

•  a  wide  variety  of  measures  based  on  differing  information  needs 
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Distribution  in  rank  order  from  low  to  high 

♦  Lower  Maturity  Organizations  (Maturity  Levels  1-3) 
a  Higher  Maturity  Organizations  (Maturity  Levels  4  and  5) 


Figure  8:  Changes  Over  Time  in  Productivity 


Table  9:  Summary  Benefits  and  Impact — Productivity 


# 

Result 

Organization 

1 

Used  Measurement  and  Analysis  process  area  to  realize  an  1 1  percent  increase 
in  productivity,  corresponding  to  $4.4M  in  additional  value 

Anonymous  2 

2 

$103  million  dollars  saved  in  maintenance  costs  and  $99  million  dollars  saved 
in  development  costs  due  to  increased  productivity  as  the  organization  moved 
from  SW-CMM  maturity  level  3  toward  CMMI  maturity  level  5 

IBM  Australia 

Application 

Management  Services 
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# 

Result 

Organization 

3 

Over  20  percent  improvement  in  account  productivity  as  the  organization 
moved  from  SW-CMM  maturity  level  3  toward  CMMI  maturity  level  5 

IBM  Australia 

Application 

Management  Services 

4 

Doubled  the  number  of  releases  from  three  to  six  per  year  as  the  organization 
implemented  CMMI  maturity  level  3 

JP  Morgan  Chase 

5 

Productivity,  measured  in  source  statements  per  month,  increased  by  1 1  percent 
between  SW-CMM  maturity  level  5  and  CMMI  maturity  level  5.  This 
improvement  is  also  a  72  percent  increase  from  SW-CMM  maturity  level  3 

Lockheed  Martin 

Maritime  Systems  & 
Sensors  -  Undersea 
Systems 

6 

Doubled  labor  productivity  as  the  organization  progressed  toward  CMMI 
maturity  level  5  from  SW-CMM  maturity  level  3 

SAIC  System  and 

Network  Solutions 

Group  (SNSG) 

7 

25  percent  productivity  improvement  using  CMMI  over  a  three  year  period  as 
the  organization  moved  from  SW-CMM  maturity  level  5  toward  CMMI 
maturity  level  5 

Siemens  Information 
Systems  Ltd. 

Expanding  on  the  statement  in  the  third  row  of  Table  9,  Figure  9  shows  the  20  percent 
increase  in  account  productivity  at  IBM  Australia  Application  Management  Services  in  more 
detail  expressed  as  function  points  per  full-time  equivalent  staff  member  (FP/FTE). 
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Figure  9:  Productivity  Improvement  at  IBM  Australia 

Expanding  on  the  statement  in  the  sixth  row  of  Table  9,  Figure  10  shows  the  pattern  over 
time  leading  up  to  a  major  improvement  in  labor  productivity  as  the  SAIC  System  and 
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Network  Solutions  Group  progressed  toward  CMMI  maturity  level  5  from  SW-CMM 
maturity  level  3. 


Labor  Productivity  T rend 


—3bo'  FYcductvcy 


Figure  10:  Productivity  Improvement  at  SAIC 

3.3.4  Quality 

Improvement  in  product  quality  is  most  frequently  measured  by  reductions  in  numbers  of 
defects.  Once  again,  the  specific  measures  vary  depending  on  the  business  objectives  and 
other  information  needs  of  the  reporting  organizations.  Such  measures  include  counts  by 
phase  of  injection  or  discovery  and  total  density. 

As  seen  in  Figure  11,  twelve  data  points  come  from  lower  maturity  organizations  that  have 
reported  improvements  over  time  in  product  quality.  As  is  true  with  the  other  measures  of 
improved  performance,  there  can  be  many  reasons  for  the  differences  between  these  lower 
maturity  organizations  and  the  greater  number  of  higher  maturity  organizations  that  have 
released  their  quality  improvement  results. 
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Distribution  in  rank  order  from  low  to  high 


♦  Lower  Maturity  Organizations  (Maturity  Levels  1-3) 
a  Higher  Maturity  Organizations  (Maturity  Levels  4  and  5) 


These  data  points  include 

•  pilot  projects  focused  on  particular  processes 

•  major  organization-wide  improvement  initiatives 

•  a  wide  variety  of  measures  based  on  differing  information  needs 


Figure  11:  Changes  Over  Time  in  Quality 

Table  10  includes  a  few  assertion  statements  that  describe  the  kinds  of  improvements  in 
product  quality  that  have  been  achieved  through  CMMI-based  process  improvement.  The 
first  table  entry  is  from  the  same  anonymous,  medium-sized  company  that  reported  cost 
savings  due  to  improved  hardware  engineering  practices  (see  the  earlier  discussion  about  the 
first  row  of  Table  7  in  Section  3.3.1).  Although  this  company  was  only  aiming  for  a  CMMI 
maturity  level  3  determination,  it  consciously  used  causal  analysis  and  resolution  practices  to 
achieve  its  44  percent  reduction  in  defects. 


Table  10:  Summary  Benefits  and  Impact — Quality 


# 

Result 

Organization 

1 

44  percent  defect  reduction  following  causal  analysis  cycle  at  an  organization 
moving  toward  CMMI  maturity  level  3 

Anonymous  1 

2 

40  percent  reduction  in  all  production  problems  and  over  80  percent  reduction 
in  Severity  1  problems,  as  the  organization  moved  from  SW-CMM  maturity 
level  3  toward  CMMI  maturity  level  5 

IBM  Australia 

Application 

Management  Services 
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# 

Result 

Organization 

3 

Reduced  defect  rate  at  CMMI  maturity  level  5  approximately  two  thirds 
compared  to  performance  at  SW-CMM  maturity  level  5 

Lockheed  Martin 

Maritime  Systems  and 
Sensors  -  Undersea 
Systems 

4 

Reduced  software-defects-per-million-delivered-SLOC  from  a  1992  baseline  by 
over  50  percent  at  SW-CMM  maturity  level  5  and  then  to  80  percent  at  CMMI 
maturity  level  5 

Lockheed  Martin 

Systems  Integration, 
Owego,  NY 

5 

Reduced  identified  defects  from  6.6  to  2.1  per  KLOC  over  three  causal  analysis 
and  resolution  cycles  in  a  CMMI  maturity  level  5  organization  with  PSP  trained 
engineers 

Northrop  Grumman  IT, 
Defense  Enterprise 
Solutions 

6 

Over  110  percent  improvement  in  effectiveness  of  phase  containment  as  the 
organization  moved  from  SW-CMM  maturity  level  3  to  CMMI  maturity  level  5 

Reuters 

7 

Improved  defect  removal  before  test  from  50  percent  to  70  percent,  leaving  0.35 
post  release  defects  per  KLOC  as  the  organization  moved  from  SW-CMM 
maturity  level  5  toward  CMMI  maturity  level  5 

Siemens  Information 
Systems  Ltd. 

8 

Reduced  defect  density  an  average  of  7 1  percent  in  three  technical  areas.  On 
average,  46  percent  of  that  reduction  occurred  as  the  organization  moved  from 
SW-CMM  maturity  level  5  toward  CMMI  maturity  level  5 

Siemens  Information 
Systems  Ltd. 

As  seen  in  Figure  12,  IBM  Australia  Application  Management  Services  achieved  a  40  percent 
reduction  in  all  production  problems,  along  with  a  reduction  of  over  80  percent  in  Severity  1 
problems  as  the  organization  progressed  toward  CMMI  maturity  level  5.  These  improvements 
in  product  quality  complement  the  20  percent  increase  in  account  productivity  shown  in 
Figure  9. 
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Figure  12:  Quality  Improvement  at  IBM  Australia 

Figure  13  shows  average  reductions  of  46  percent  in  defect  density  at  Siemens  Information 
Systems  Ltd.  as  it  moved  from  SW-CMM  maturity  level  5  toward  CMMI  maturity  level  5. 
Comparable  improvements  in  cost  of  quality  and  estimation  accuracy  are  noted  for  the  same 
organization  in  Table  7.  Productivity  improvements  during  the  same  time  period  are  found  in 
Table  9. 
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Figure  13:  Quality  Improvement  at  Siemens 

Finally,  Figure  14  shows  the  marked  reductions  in  defects  achieved  at  Northrop  Grumman  IT. 
They  come  from  the  same  three-and-a-half-year,  firm-fixed  price  inventory  tracking  project 
that  achieved  comparable  gains  in  cost  and  schedule  performance  (see  Figure  4  and  Figure  7). 
Although  the  defect  density  rose  slightly  immediately  after  the  third  causal  analysis  cycle,  it 
fell  substantially  after  the  fifth  build. 
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Defect  prevention  using  PSP  and  CAR  at  CMMI  MLS 


Figure  14:  Quality  Improvement  at  Northrop  Grumman  IT 

3.3.5  Customer  Satisfaction 

This  category  generally  includes  changes  based  on  customer  surveys.  Award  fees  also  are 
sometimes  used  as  surrogate  measures.  As  seen  in  some  of  the  more  detailed  case 
descriptions  that  follow  in  Section  4,  some  organizations  do  regularly  collect,  analyze,  and 
use  quantitative  measures  of  customer  satisfaction.  Few  results  that  can  be  expressed  as 
change  over  time  are  available  yet.  Only  one  of  them  comes  from  a  maturity  level  3 
organization  that  already  was  on  its  way  to  CMMI  maturity  level  5.  This  is  why  no  data 
points  from  lower  maturity  organizations  are  included  in  Figure  15. 
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Distribution  in  rank  order  from  low  to  high 


▲  Higher  Maturity  Organizations  (Maturity  Levels  4  and  5) 


These  data  points  include 

•  pilot  projects  focused  on  particular  processes 

•  major  organization-wide  improvement  initiatives 

•  a  wide  variety  of  measures  based  on  differing  information  needs 


Figure  15:  Changes  Over  Time  in  Customer  Satisfaction 

With  the  exception  of  those  that  are  described  in  more  detail  in  the  case  descriptions  in 
Section  4,  Table  1 1  includes  all  of  the  examples  that  can  be  expressed  as  change  over  time. 
The  table  also  includes  a  few  examples  that  cannot  be  expressed  as  change  over  time. 


Table  11:  Summary  Benefits  and  Impact — Customer  Satisfaction 


# 

Result 

Organization 

1 

Slight  decline  in  customer  satisfaction  as  the  organization  moved  from  SW- 
CMM  maturity  level  4  to  CMMI  maturity  level  5.  The  results  remain  essentially 
stable  for  a  series  of  customer  surveys,  with  average  survey  responses  around  4 
on  a  five  point  scale  over  the  entire  time  period. 

The  Boeing  Company 

2 

Increased  award  fees  by  55  percent  compared  to  an  earlier  SW-CMM  baseline 
at  maturity  level  2 

Lockheed  Martin 
Management  and  Data 
Systems 

3 

Earned  an  exceptional  rating  in  every  applicable  category  on  a  contractor 
performance  evaluation  survey  in  a  CMMI  maturity  level  5  organization 

Northrop  Grumman  IT, 
Defense  Enterprise 
Solutions 

4 

Received  more  than  98  percent  of  possible  customer  award  fees  in  a  CMMI 
maturity  level  5  organization 

Northrop  Grumman  IT, 
Defense  Enterprise 
Solutions 
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# 

Result 

Organization 

5 

Steady  increase  in  customer  satisfaction  as  the  organization  progressed  toward 
CMMI  maturity  level  5  from  SW-CMM  maturity  level  3 

SAIC  System  and 

Network  Solutions 

Group 

6 

Customer  satisfaction  index  increased  an  average  of  42  percent  in  three 
technical  areas  from  1996  through  2004.  An  average  of  48  percent  of  that 
improvement  occurred  as  the  organization  moved  from  SW-CMM  maturity 
level  5  toward  CMMI  maturity  level  5 

Siemens  Information 
Systems  Ltd. 

As  seen  in  the  fifth  row  of  the  table,  there  was  a  steady  increase  in  customer  satisfaction  at 
SAIC;  further  detail  can  be  seen  in  Figure  16.  Note  in  the  sixth  row  of  the  table  that  there 
were  improvements  in  three  technical  areas  at  Siemens.  The  MP  and  Telecom  results  are 
virtually  the  same,  so  they  are  indistinguishable  in  Figure  17. 
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Figure  16:  Customer  Satisfaction  at  SAIC 
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Figure  1 7:  Customer  Satisfaction  at  Siemens 

3.3.6  Return  on  Investment 

Return  on  investment  (ROI)  can  be  expressed  in  many  ways  [Denne  04,  El  Emam  05].  In 
addition  to  benefit-to-cost  ratios,  these  include  companion  measures  of  net  present  value, 
internal  rate  of  return,  payback  periods,  and  break  even  points.  More  comprehensive 
benchmarking  comparisons  will  need  to  consider  all  of  these. 

The  results  reported  here  include  only  those  that  can  be  expressed  as  ratios  of  benefits  to 
costs  over  time.  As  is  true  for  all  of  the  other  performance  categories  in  this  technical  report, 
the  costs  and  benefits  included  here  vary  according  to  the  business  objectives  and  other 
information  needs  of  the  reporting  organizations. 

Many  higher  maturity  organizations  apparently  calculate  return  on  investment  or  do  related 
cost-benefit  studies,1  and  very  few  of  the  ROI  results  reported  here  come  from  lower 
maturity  organizations  (see  Figure  18).  Similarly,  the  scope  of  the  efforts  for  which  these 
organizations  report  ROI  results  tends  to  focus  on  specific  process  improvements  in 
individual  business  units.  Reports  of  return  on  total  improvement  investment  are  much  less 
common  for  larger  organizational  unit  or  enterprise-wide  maturity  level  improvement 


1  In  a  recent  survey,  64  percent  of  the  respondents  reported  that  they  calculate  ROI  or  do  related 
cost-benefit  studies.  The  majority,  but  not  all,  of  the  59  respondents  were  from  higher  maturity 
organizations.  These  results  are  from  an  unpublished  report  by  D.  Goldenson,  G.  Draper,  and  D. 
Gibson  titled  Benchmarking  CMMI  Cost  and  Impact:  Interim  Report.  This  report  was  completed 
in  December  2004,  but  the  distribution  of  the  full  document  is  limited  to  the  benchmark 
contributors. 
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initiatives.  That  is  why  there  is  so  much  variation  in  the  ROI  ratios  in  Figure  18  and  Table  12 
and  why  some  of  them  may  appear  to  be  so  high. 


Distribution  in  rank  order  from  low  to  high 

♦  Lower  Maturity  Organizations  (Maturity  Levels  1-3) 

▲  Higher  Maturity  Organizations  (Maturity  Levels  4  and  5) 


These  data  points  include 

•  pilot  projects  focused  on  particular  processes 

•  major  organization-wide  improvement  initiatives 

•  a  wide  variety  of  measures  based  on  differing  information  needs 


Figure  18:  Changes  Over  Time  in  ROI 

Higher  maturity  organizations  in  particular  appear  to  be  much  more  interested  in  reducing 
costs  of  quality  and  poor  quality,  and  in  improving  product  quality  and  customer  satisfaction, 
than  they  are  in  overall  ROI.  Instead,  some  organizations  use  estimates  of  ROI  to  decide  how 
to  allocate  their  available  process  improvement  resources. 

Table  12  includes  a  few  selected  assertion  statements  that  describe  the  kinds  of  ROI 
improvements  that  have  been  achieved  through  CMMI-based  process  improvement.  Again, 
further  information  can  be  found  in  the  more  detailed  case  descriptions  in  Section  4. 

The  first  three  entries  in  Table  12  have  reported  overall  return  on  their  investments  aimed  at 
maturity  level  improvements.  The  fourth  entry,  from  Tata  Consultancy  Services,  describes  the 
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overall  return  on  their  process  improvement  initiatives  for  an  entire  year.2  The  rest  of  the 
table  entries  describe  the  ROI  of  more  focused  process  improvement  efforts. 

The  fifth  table  entry  is  from  the  same  anonymous  organization  that  used  the  CMMI 
Measurement  and  Analysis  process  area  to  structure  its  initial  CMM-based  process 
improvement  effort.  Not  only  did  this  organization  reduce  its  cost  of  quality  and  improve  its 
productivity  markedly  in  the  space  of  one  year,  it  also  realized  a  notable  return  on  its 
improvement  investment,  even  when  amortized  over  less  than  six  months. 

Finally,  there  was  a  very  high  return  on  investment  achieved  at  Northrop  Grumman 
Information  Technology,  Defense  Enterprise  Solutions.  Once  again,  this  result  is  from  the 
same  inventory  tracking  project  discussed  earlier  that  went  through  three  causal  analysis  and 
resolution  cycles  to  improve  its  peer  review  processes  (see  Figure  4,  Figure  7,  and  Figure  14). 
As  can  be  seen  in  more  detail  on  the  CMMI  Performance  Results  Web  site,  investments  for 
calculating  the  13:1  ROI  included  the  effort  to  conduct  the  causal  analysis  cycles,  refine  the 
process  assets,  and  develop  and  deliver  additional  team  training.  Benefits  were  calculated  by 
estimating  the  effort  and  cost  that  would  have  been  incurred  had  the  defect  density  remained 
at  the  build  1  baseline. 


Table  12:  Summary  Benefits  and  Impact — Return  on  Investment 


# 

Result 

Organization 

1 

6:1  ROI  in  a  CMMI  maturity  level  3  organization 

Raytheon  Corporation, 
Anonymous  site 

2 

3:1  ROI  in  a  CMMI  maturity  level  5  organization 

Raytheon  Network 

Centric  Systems,  North 
Texas 

3 

2:1  ROI  over  3  years  as  the  organization  moved  from  SW-CMM  maturity  level 

5  toward  CMMI  maturity  level  5 

Siemens  Information 
Systems  Ltd. 

4 

ROI  of  2.65:1  from  Organizational  Innovation  and  Deployment  (and  People 
CMM)  improvements  across  all  Development  Centers  in  a  CMMI  maturity 
level  5  organization 

Tata  Consultancy 

Services 

5 

Used  Measurement  and  Analysis  process  area  to  realize  a  2.5:1  ROI  over  1st 
year  prior  to  being  assessed  at  SW-CMM  maturity  level  2,  with  benefits 
amortized  over  less  than  6  months 

Anonymous  2 

6 

13:1  ROI,  calculated  as  defects  avoided  per  hour  spent  in  training  and  defect 
prevention,  over  three  causal  analysis  and  resolution  cycles  in  a  CMMI  maturity 
level  5  organization  with  PSP  trained  engineers 

Northrop  Grumman  IT, 
Defense  Enterprise 
Solutions 

7 

Over  2.5  ROI  from  process  automation  estimated  at  maturity  level  5 

Reuters 

8 

Over  3:1  ROI  due  to  post  release  defect  reduction  as  the  organization  moved 
from  SW-CMM  maturity  level  3  to  CMMI  maturity  level  5 

Reuters 

9 

ROI  of  4.28:1  realized  by  reduced  rework  effort  through  early  detection  of 
memory  leaks  in  a  CMMI  maturity  level  5  organization 

Tata  Consultancy 

Services 

10 

ROI  of  5.21:1  in  productivity  enhancement,  defect  reduction,  and  shorter 
delivery  cycle  time  by  implementing  and  utilizing  a  Center  of  Excellence  Web 
site  in  a  CMMI  maturity  level  5  organization 

Tata  Consultancy 

Services 

2  Tata  also  estimates  ROI  as  part  of  its  regular  process  management  activities,  which  closely  follow 
the  guidance  in  the  CMMI  Organizational  Innovation  and  Deployment  process  area. 
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# 

Result 

Organization 

11 

ROI  of  5.33:1  realized  through  the  development  and  utilization  of  use  case 
estimation  guidelines  and  a  template  in  a  CMMI  maturity  level  5  organization 

Tata  Consultancy 

Services 

12 

ROI  of  27.7:1  in  productivity  enhancement,  defect  reduction,  and  shorter 
delivery  cycle  time  realized  by  implementing  and  utilizing  an  Asset  Repository 
at  Centers  of  Excellence  in  the  Development  Centres  in  a  CMMI  maturity  level 

5  organization 

Tata  Consultancy 

Services 
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4  Case  Descriptions 


The  following  case  descriptions  provide  additional  detail  about  the  performance  results 
achieved  by  10  organizations  that  have  used  CMMI  models  to  guide  their  process 
improvement  efforts.  The  results  include  examples  for  all  six  performance  categories 
described  earlier  in  this  technical  report.  Each  description  provides  background  information 
about  the  participating  organization  and  its  products  and  prior  experience  with  process 
improvement.  The  background  information  is  followed  by  additional  detail  about  the 
organization’s  implementation  of  practices  advocated  by  CMMI  models.  The  performance 
results  are  presented  along  with  additional  detail  about  business  objectives,  measurement 
criteria,  and  actions  that  were  taken  based  on  the  analyses. 

Some  of  the  results  have  been  normalized  to  protect  proprietary  information.  SEI  staff 
members  reviewed  the  data  and  results  to  better  understand  them  and  judge  their  accuracy.  In 
several  instances,  SEI  staff  also  provided  assistance  with  study  design  and  analysis. 
Individuals  from  participating  organizations  were  asked  to  provide  their  evidence  and 
contextual  information  using  a  standard  format  to  facilitate  summarization  and  quality 
assurance.  They  were  asked  to  discuss  their  measurement  criteria,  including  evidence  that  the 
results  can  properly  be  attributed  to  CMMI -based  processes  as  opposed  to  other  factors  or 
unintended  measurement  effects. 

4.1  3H  Technology 

written  with  Ruth  Buys 


Background 

Organization  Background 

3H  Technology  (3HT)  is  an  information  technology  company  offering  a  wide  range  of 
products  and  technical  services.  These  include  performing  custom  software  development, 
systems  integration,  and  product  implementation.  3HT  develops  custom-tailored  applications 
for  the  automotive,  medical,  pharmaceutical,  real  estate,  property  management,  banking  and 
finance,  and  retail  industries.  Product  implementation  services  include  analysis  of  user 
environments,  followed  by  custom  tailoring  of  its  products.  Other  services  include  providing 
quality  assurance,  network  support  and  training,  writing  technical  and  user  documentation, 
and  offering  process  improvement  consulting.  3HT  distributes  two  of  their  own  products: 

•  eSnap — a  knowledge  management  program 
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•  ElectroLease — a  document  imaging  software  package  that  was  built  to  meet  the  special 
interests  of  real  estate  professionals. 

3HT  has  about  185  full-time  staff  and  20  to  25  consultants.  Approximately  60  full-time  staff 
work  in  Network  Solutions;  the  remaining  staff  work  in  Software  Development  or  Program 
Management.  The  Network  Solutions  staff  supports  over  2,100  users  in  over  67  sites  around 
the  Washington,  D.C.  Metropolitan  area. 

Process  Improvement  History 

3HT  had  no  real  experience  with  process  improvement  prior  to  starting  its  ISO  and  quality 
assurance  initiatives  in  2004.  The  quality  assurance  initiative  began  in  the  Software 
Development  Division,  and  the  ISO  9001:2000  certification  effort  began  in  the  Network 
Solutions  Division.  A  few  months  later,  but  before  ISO  certification  was  achieved,  the 
Software  Development  and  Program  Management  Divisions  began  work  with  CMMI.  During 
the  period  from  March  2004  through  December  2004,  quality  assurance,  ISO,  and  CMMI 
initiatives  were  proceeding  concurrently.  In  January  2005,  the  ISO  effort  went  into 
maintenance  mode. 


CMMI-Based  Process  Improvement 

3HT  began  its  CMMI  implementation  essentially  from  scratch.  During  the  first  four  weeks  of 
the  CMMI  effort,  a  process  improvement  infrastructure,  comprised  of  an  Engineering  Process 
Group  (EPG)  and  a  Management  Steering  Group  (MSG),  was  established.  3HT  had  strong 
management  support  from  its  senior  management,  as  evidenced  by  the  willing  attendance  of 
the  CEO,  COO,  and  CIO  at  a  two-day  training  session  on  CMMI.  It  took  some  time  to 
establish  the  management  steering  committee  and  to  understand  what  was  needed.  The  senior 
managers  agreed  to  participate  on  the  management  steering  committee,  attended  all  of  the 
meetings,  and  responded  to  requests.  While  there  was  some  reluctance  among  individuals  in 
middle  management,  3HT  was  finally  able  to  garner  their  support  as  well. 

The  process  improvement  strategy  undertaken  at  3HT  was  not  unusual.  3HT  began  with  the 
initiation  of  several  Process  Action  Teams  (PATs)  who  worked  to  establish  guidance  on  how 
to  document  processes  and  other  documentation  conventions.  The  effort  initially  focused 
solely  on  reducing  defects  in  delivered  software.  Then,  work  started  in  the  Project 
Management  area  with  the  review  of  existing  assets  and  a  gap  analysis  against  the  relevant 
CMMI  process  areas.  These  included  Project  Planning  (PP),  Project  Monitoring  and  Control 
(PMC),  and  Risk  Management  (RSKM).  For  the  most  part,  3HT  was  able  to  build  on  existing 
processes.  However,  there  were  several  significant  process  elements  that  3HT  had  to  create 
from  the  beginning.  These  included  two  example  plans,  a  general  project  plan  and  a  quality 
assurance  plan,  and  a  process  for  formal  risk  identification  and  tracking.  Additionally,  the 
software  group  had  to  adopt  a  standard  form  for  preparing  its  project  plans  and  the  associated 
work  breakdown  structures.  The  initial  SCAMPIsm  Class  C  appraisal  was  completed  while 
the  Project  Management  PAT  was  completing  its  work.  The  information  from  the  appraisal 
was  used  to  validate  the  revised  project  management  process  descriptions. 
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Other  process  action  teams  were  chartered  in  succession  until  all  CMMI  maturity  level  2  and 
3  process  areas  were  addressed.  Improvements  were  made  to  processes  which  mapped  to 
Requirements  Management  (REQM),  Configuration  Management  (CM),  Project  and  Product 
Quality  Assurance  (PPQA),  and  Measurement  and  Analysis  (MA).  New  processes  begun  as 
part  of  the  process  improvement  initiative  included  Validation  (VAL),  Verification  (VER), 
and  Project  and  Product  Quality  Assurance  (PPQA)  for  all  software  development  projects. 
Improvements  included  tracking  size  data,  adding  a  requirements  traceability  matrix,  and 
launching  configuration  audits.  New  organization-level  processes  included  the  establishment 
and  monitoring  of  a  process  asset  library  and  the  collection  and  reporting  of  process 
improvement  measures.  3HT  documented  and  integrated  existing  processes  where  possible 
and  defined  new  processes  where  it  was  not.  Since  many  of  the  existing  processes  at  the 
organizational  level  as  well  as  the  project  level  had  not  existed  on  paper  before  this  time,  and 
were  not  used  consistently,  all  of  the  defined  processes  seemed  new. 

All  of  the  previously  described  process  documentation  took  place  while  the  EPG  was 
working  intensely  with  two  pilot  projects.  In  August  2005,  3HT  sponsored  an  externally  led 
SCAMPI  Class  A  appraisal  and  achieved  CMMI  maturity  level  3. 

After  the  SCAMPI  appraisal,  3HT  continued  deploying  the  newly  documented  and  defined 
processes  on  all  software  development  projects  and  validated  the  use  of  these  processes.  3HT 
continues  to  perform  regular  gap  analyses,  comparing  the  processes  used  in  the  projects  to  the 
documented  processes.  Every  project  that  lasts  for  at  least  six  months  will  experience  at  least 
one  full  and  one  follow-up  audit.  The  Quality  Assurance  and  Process  Improvement  groups 
plan  these  activities  for  each  quarter.  Since  August  2005,  all  improvement  processes  are 
monitored  and  tracked  by  the  EPG  to  ensure  continued  adherence  to  relevant  standards, 
implementation  by  all  projects,  and  continuous  improvement. 


Performance  Results 

Performance  measurement  related  activities  in  3HT  include  the  collection  and  reporting  of 
process  improvement  measures  such  as  the  level  of  effort,  the  size  of  the  process  asset 
repository,  the  number  of  non-conformances  identified  in  process  audits,  and  the  progress  on 
deliverables.  This  data  is  reported  regularly  to  senior  management. 

Quantitative  results  provide  interesting  insights  into  the  process  improvement  effort  at  3HT. 
Some  of  the  results  still  raise  questions  on  the  sources,  validity,  and  usefulness  of  the  data 
3HT  is  collecting.  While  there  is  sufficient  data  to  conduct  analyses  relative  to  several 
process  areas,  the  data  is  not  consistent  in  all  cases.  Some  of  the  data  is  subject  to 
interpretation,  and  the  different  timeframes  in  which  the  data  was  captured  sometimes  make 
it  difficult  to  perform  comparisons.  These  difficulties  are  common  and  can  be  expected  in 
many  organizations  initiating  a  measurement  program. 
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Measuring  Software  Defects 

One  of  the  reasons  3HT  started  process  improvement  was  to  address  problems  related  to 
defects  in  the  software.  3HT  needed  to  understand  the  level  of  quality  in  software  delivered 
to  the  customer.  The  quality  assurance  process  at  3HT  transfers  software  from  the  developers, 
who  are  using  tools  to  track  defects,  to  the  quality  control  group,  where  Rational’s 
ClearQuest  is  employed.  After  management  authorization,  the  software  is  delivered  to  the 
customer.  The  customer  then  sends  the  software  to  an  independent,  third-party  contractor 
who  does  verification  and  validation  (IV&V).  Data  is  gathered  to  show  the  number  of  defects 
collected  during  quality  control  tests  using  ClearQuest  compared  to  the  average  number  of 
defects  identified  per  release  from  IV&V  reports.  If  errors  are  found  in  IV&V  testing,  3HT 
does  the  rework. 

Prior  to  process  intervention,  the  releases  were  on  average  75  percent  defect  free.  3HT  took  a 
look  at  the  effectiveness  of  its  quality  control  procedures  by  comparing  the  average  number 
of  defects  found  before  and  after  implementing  these  procedures.  The  organization  found  that 
after  the  process  intervention  the  releases  are  on  average  98  percent  defect  free. 


Figure  19:  Number  of  Defects 

The  results  seen  in  Figure  19  can  be  attributed  to  CMMI  because  while  3HT  had  a  method  to 
identify  and  record  defects  from  IV&V  reports  before  the  CMMI-based  quality  assurance, 
validation,  and  verification  processes  were  implemented,  the  analysis  of  that  data,  which 
could  have  helped  3HTdecide  what  to  improve,  was  not  directly  supported. 
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Measuring  Process  Improvement 

3HT  captured  data  about  the  costs  of  doing  CMMI-based  process  improvement  inside  the 
organization.  Originally,  the  organization  estimated  it  would  take  a  total  of  13,000  hours  to 
perform  the  process  improvement  work.  It  took  less  time  than  that  for  several  reasons. 

First,  the  efforts  associated  with  process  improvement  tasks  were  not  tracked  during  the  first 
four  months  of  the  pilot  phase  because  it  took  that  long  to  establish  and  deploy  a  viable  effort 
measurement  process  and  get  the  appropriate  accounts  established  in  the  Finance  department. 
The  need  to  establish  new  accounts  in  Finance  brought  to  light  the  lack  of  defined  procedures 
for  completing  and  approving  time  cards.  When  the  EPG  noted  this,  the  Finance  and  Human 
Resources  departments  requested  assistance  from  the  EPG  in  defining  its  processes.  The 
assistance  was  supplied  in  the  areas  that  were  immediately  needed  by  the  EPG,  and  the 
problems  were  reported  to  senior  management,  so  additional  actions  could  be  taken,  if  they 
were  deemed  necessary.  It  also  took  less  time  than  expected  to  do  process  improvement 
because  the  EPG  followed  a  disciplined  improvement  method  and  they  recognized  the  benefit 
of  defining  and  improving  existing  processes  wherever  possible  rather  than  building  new 
processes. 

Accurate  effort  reporting  allowed  3HT  to  track  time  spent  on  various  work  components,  to 
compare  estimates  to  actuals,  and  to  calculate  other  performance  measures  such  as 
cost/benefit  analysis  and  earned  value.  Other  measures  that  were  established  included  the  size 
of  the  process  asset  repository,  number  of  non-conformances  identified  in  process  audits,  and 
progress  on  deliverables.  These  measures  were  used  for  many  processes,  but  measurement 
began  with  the  process  improvement  effort. 

3HT  tracked  the  estimated  versus  the  actual  hours  spent  on  process  improvement  tasks. 
Estimates  were  logged  into  a  spreadsheet,  and  the  actual  hours  spent  were  transferred  from 
time  sheets  to  this  spreadsheet  so  that  they  could  be  compared.  Because  of  the  initial 
improvement  efforts  initiated  with  the  Finance  Department,  3HT  was  able  to  use  the  data 
from  Finance  Department  reports.  The  original  level  of  effort  estimated  was  13,000  hours. 

The  actual  level  of  effort  for  the  12-month  period,  covering  the  last  8  months  of  the  pilot 
projects  and  the  first  4  months  of  the  rollout,  was  approximately  8,000  hours.  This  includes 
the  time  spent  training  approximately  100  staff  members. 

Measuring  Process  Enactment  and  Compliance 

As  with  all  organizations  improving  processes,  it  became  important  to  track  the  changes 
being  made,  and  because  CMMI  was  being  followed  it  was  desirable  to  map  these  changes  to 
the  model.  The  following  measures  were  taken  to  track  the  organization’s  progress  toward 
CMMI  maturity  levels  (Table  13).  First,  the  number  of  practices  that  could  be  categorized  as 
largely,  partially,  or  not  implemented  were  captured.  Then,  a  percentage  of  compliance  to 
CMMI  process  area  practices  was  calculated  and  displayed  using  a  stoplight  chart.  Prior  to 
the  process  improvement  effort,  the  implementation  of  the  practices  were  as  follows: 
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Table  13:  Implementation  of  Processes  Prior  to  Improvement  Effort 


Level  2  practices 

Largely  implemented 

Green 

25% 

Partially  implemented 

Yellow 

50% 

Not  implemented 

Red 

25% 

Level  3  practices 

Largely  implemented 

Green 

27% 

Partially  implemented 

Yellow 

43% 

Not  implemented 

Red 

30% 

Figure  20  shows  the  same  results  in  graphic  form. 


CM  MI  SCAM  PI  Class  C  Process 
Compliance  Results  (Specific 
Practices)  January  2005 


100% 

80% 

60% 


Red  Yellow  Green 


Level  2 
Level  3 


Figure  20:  SCAMPI  Class  C  Results 

After  the  process  improvement  effort  and  the  SCAMPI  Class  A  appraisal  in  2005,  all  of  the 
practices  for  which  3HT  was  appraised  were  categorized  100  percent  Green.  This  included  all 
of  the  practices  in  the  process  areas  at  maturity  level  2  and  maturity  level  3  except  Supplier 
Agreement  Management,  which  was  excluded  from  the  scope  of  the  improvement  effort  and 
SCAMPI  appraisals. 

In  order  to  evaluate  the  level  of  compliance  with  internal  organizational  standard  processes, 
processes  are  evaluated  through  internal  process  audits.  The  numbers  of  major  and  minor 
process  non-conformances  are  collected  and  compared  to  the  expected  values  for  each 
process  area.  In  the  first  half  of  2005,  66  instances  of  non-conformance  were  identified;  after 
process  intervention,  in  the  second  half  of  2005,  52  instances  of  non-conformance  were 
identified.  EPG  members  believe  that  the  number  of  non-conformances  should  have  gone 
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down  more,  but  they  are  still  improving  the  effectiveness  of  the  audit  process.  Where 
instances  of  non-conformance  were  found,  members  of  the  EPG  worked  with  project 
managers  to  develop  corrective  actions. 

The  EPG  also  examined  its  processes  by  keeping  track  of  the  number  of  process  change 
requests  (PCRs)  it  received.  In  the  first  pass,  after  implementing  new  processes,  the 
organization  received  58  PCRs.  3HT  used  these  to  identify  and  track  input  on  process 
performance,  to  determine  process  volatility,  and  to  support  continuous  improvement.  PCRs 
are  entered  into  a  Process  Change  Request  Log  that  is  then  reviewed  by  subject  matter 
experts  for  an  assessment  of  the  impact  and  generation  of  recommendations. 

3HT  intends  to  maintain  what  it  has  gained  from  process  improvement  and  plans  to 
implement  continuous  improvements.  The  organization  is  considering  achieving  higher 
maturity  levels  as  time  and  resources  permit. 


4.2  ABB 

written  with  Aldo  Dagnino 

Background 

Organization  Background 

ABB  is  a  leader  in  power  and  automation  technologies.  It  enables  utility  and  industry 
customers  to  improve  performance  while  lowering  environmental  impact.  ABB’s  products 
help  operate  utilities,  process  industries,  manufacturing  plants,  and  other  industries.  ABB  has 
representation  in  over  120  countries  and  employs  1 10,000  people.  A  vast  majority  of  ABB’s 
products  have  software  and  hardware  components. 

The  ABB  unit  presented  in  this  case  description  develops  modular  substation  automation 
systems  to  monitor  the  condition  of  electrical  equipment  at  electrical  utility  companies.  This 
equipment  includes  circuit  breakers,  power  transformers,  and  other  transmission  and 
distribution  systems.  ABB  substation  automation  systems  provide  remote  control  and 
monitoring  capabilities  of  all  kinds,  from  power  distribution-level  to  extra-high  voltage 
substations.  The  substation  automation  systems  developed  by  this  ABB  unit  are  quite 
complex  and  include  hardware  and  software  components.  The  products  make  use  of 
intelligent  electronic  devices  (IEDs)  for  protection  and  control,  and  they  lay  the  foundation 
for  all  the  higher  level  functions  such  as  advanced  power  system  management  and  the 
monitoring  of  the  condition  of  the  equipment  while  it  is  in  use.  The  software  that  is 
embedded  in  the  hardware  of  substation  automation  products  makes  the  systems  easy  to  use 
and  adaptable  to  customer-specific  requirements.  The  scalable  modular  systems  developed  by 
this  unit  reflect  the  typical  utility  customer  needs  for  the  following  range  of  applications: 

•  transmission,  sub-transmission  and  distribution  substations 
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•  new  installations  and  refurbishment  of  existing  substations 

•  utilities  and  industries 

•  gas  and  air  isolated  switchgear 

A  sample  product  that  shows  the  complexity  of  substation  automation  systems  developed  by 
this  unit  is  the  substation  automation  system  for  sub-transmission  and  distribution  of  power 
(SAS510).  The  SAS510  contains  a  station  computer  with  human  machine  interface  (HMI)  for 
local  control,  monitoring,  and  system  configuration.  The  station  computer  also  is  equipped 
with  local  area  network  (LON)  interfaces  for  the  integration  of  bay  control  IEDs  and  with 
LON  and/or  IEC60870-5-103  interfaces  for  the  integration  of  protection  IEDs. 

Process  Improvement  History 

ABB  starting  using  the  SW-CMM  for  process  improvement  in  some  of  its  units  in  1999.  A 
pilot  of  processes  based  on  CMMI  was  completed  in  the  Fall  of  2001  for  one  of  these  units. 
The  unit  described  in  this  case  never  used  SW-CMM  and  started  using  CMMI  in  January  of 
2003. 

The  ABB  software  and  systems  process  initiative  process  improvement  (ASPI)  group 
addresses  processes  for  the  full  product  life  cycle  (systems  and  software  process  initiatives). 
Since  the  organization  is  spread  across  the  globe,  there  is  a  unit  at  each  location  that  is 
responsible  for  supporting  local  process  improvement. 

ABB  has  used  the  IDEALSM  model  since  1999.  All  process  improvement  initiatives  at  ABB 
use  an  interpretation  of  IDEAL’s  continuous  process  improvement  engine.  This  is  true  not 
only  for  CMMI  but  also  for  all  ABB  efforts  based  on  principles  of  lean  manufacturing, 
business  process  reengineering,  and  the  theory  of  constraints.  This  group  uses  the  initiation 
phase  of  IDEAL  to  explain  CMMI,  ask  a  business  unit  about  its  business  objectives,  and  do  a 
quick  assessment  to  see  if  the  unit  is  ready  for  process  improvement.  A  full  IDEAL  cycle 
lasts  about  a  year  and  a  half. 

CMMI-Based  Process  Improvement 

The  following  CMMI-based  process  improvement  efforts  apply  to  one  ABB  unit  over  a  two- 
year  period  from  January  2003  to  the  end  of  2004.  The  unit  did  not  have  prior  experience 
using  CMMI,  but  there  were  notable  improvements  by  the  second  year.  This  unit  had  at  the 
time  of  the  study  approximately  35  hardware  and  software  developers. 

In  January  of  2003,  the  ABB  Corporate  Engineering  Process  Group  (CEPG)  performed  an 
internal  ABB  CMMI  Class  B  appraisal  on  this  unit  based  on  CMMI  maturity  level  2  process 
areas  and  the  Requirement  Development  and  Verification  process  areas.  This  appraisal 
resulted  in  the  identification  of  strengths  and  weaknesses.  Before  the  appraisal,  a  meeting 
with  this  unit’s  senior  management  established  that  a  major  business  objective  was  to  reduce 
the  cost  of  poor  quality  (COPQ)  by  25  percent  in  2003  as  compared  to  its  results  in  2002.  At 


38 


CMU/SEI-2006-TR-004 


that  time,  COPQ  was  defined  as  the  cost  associated  with  fixing  defects  in  the  software 
modules  after  system  level  testing  was  performed  and  before  the  product  is  shipped  to  the 
customers.  In  order  to  maintain  an  acceptable  level  of  consistency,  it  was  important  that  there 
were  no  major  differences  on  how  the  COPQ  was  determined  between  2002  and  2003. 

Using  the  results  of  the  appraisal  and  other  discoveries,  this  unit  created  an  action  plan  to 
address  all  weaknesses  identified  in  its  implementation  of  each  of  the  process  areas.  The 
business  goal  of  reduction  in  COPQ  was  used  to  prioritize  the  unit’s  process  improvement 
activities.  It  became  clear  that  in  order  to  meet  this  goal,  the  unit  needed  to  focus  on  the 
CMMI  Requirements  Development  and  Requirements  Management  process  areas.  These  two 
process  areas  were  clustered  into  one  process  at  this  ABB  unit:  the  requirements  engineering 
(RE)  process.  A  project  to  improve  the  RE  process  was  established  and  in  two  months  the 
new  process  was  developed  for  the  whole  organization  based  on  the  pertinent  CMMI 
practices.  This  process  was  piloted  and  used  at  a  later  stage  in  2003  in  a  major  product 
development  project.  The  Measurement  and  Analysis  process  area  was  employed  to  monitor 
and  determine  the  COPQ  for  the  remainder  of  2003.  After  the  pilot,  the  new  RE  process  was 
institutionalized  for  the  whole  product  development  organization  so  that  all  new  product 
development  projects  would  follow  the  same  process. 

The  appraisal  had  established  that  the  largest  source  of  defects  in  the  process  were  due  to  the 
development  and  maintenance  of  software  requirements.  Hence,  defects  were  tracked  when 
the  system-level  testing  was  performed  and  then  the  effort  and  cost  associated  with  reworking 
those  defects  were  recorded.  A  comparison  with  the  same  effort  and  cost  of  defects  recorded 
at  the  system-level  testing  in  2002  was  performed. 

The  following  points  describe  the  improvement  in  processes  for  each  CMMI  process  area  that 
was  related  to  the  organization’s  RE  process: 

•  Requirements  Development — The  ABB  unit  used  market  requirement  specifications 
(MRS)  in  the  past,  but  the  MRS  were  not  formally  defined  and  not  enough  time  was 
given  to  generate  them.  Requirements  Development  helped  this  unit  create  a  template  for 
the  elicitation  and  collection  of  requirements  and  a  process  to  consistently  analyze, 
prioritize,  and  validate  customer  requirements. 

•  Requirements  Management — This  process  area  helped  this  unit  create  a  process  to 
ensure  the  proper  understanding  of  requirements  and  set  up  a  process  to  adequately  and 
consistently  manage  changes  to  requirements.  With  these  processes  team  members  can 
come  to  a  common  consensus  about  changes;  formerly,  they  had  a  loosely  defined 
baseline  that  could  be  changed  without  a  formal  change  request  process,  and  this  would 
cause  inconsistencies  and  incompatibilities. 

•  Measurement  and  Analysis — This  ABB  unit  already  had  its  business  objectives  mapped 
to  high  level  measurement  objectives.  While  this  was  atypical  elsewhere  in  ABB,  they 
needed  more  guidance  about  how  to  choose  the  right  measures  and  analytic  techniques 
when  allocating  the  measurement  objectives  to  specific  product  development  activities. 
The  Measurement  and  Analysis  process  area  helped  them  do  this. 
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Performance  Results 


As  noted  in  the  previous  section,  this  ABB  unit  aimed  to  reduce  its  cost  of  poor  quality 
(COPQ)  by  25  percent  after  implementing  its  new  CMMI-based  requirements  engineering 
process.  The  unit  calculated  COPQ  by  measuring  defects  found  at  the  integration  and  system 
level  testing  stages  (for  software  and  hardware)  and  tracking  time  for  defect  corrections  and 
retesting.  At  this  stage,  defects  are  more  costly  to  fix.  If  defects  are  found,  the  appropriate 
components  must  be  corrected,  and  they  must  then  be  sent  back  to  hardware  or  software 
testing  and  the  time  to  fix  and  re-run  tests  must  be  tracked. 

Figure  21  shows  a  comparison  of  the  cost  of  poor  quality  in  2002  and  2003  on  a  cumulative 
basis  as  well  as  month-by-month.  The  absolute  values  have  been  sanitized  in  the  figure; 
however,  the  cumulative  cost  of  poor  quality  was  30  percent  lower  in  2003  in  the  unit’s 
second  year  of  CMMI-based  improvement. 


Reduction  in  COPQ  Associated  with  Improving  Requirements  Engineering  Processes 
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Figure  21:  Reduction  in  COPQ  Associated  with  Improving  Requirements 
Engineering  Processes 

ABB  started  calculating  ROI  corporate  wide  in  2003,  and  the  typical  ROI  for  major  process 
changes  is  3:1  to  5:1  (benefit  to  cost  of  process  improvement  activities).  This  ABB  unit  had  a 
net  economic  benefit  of  50  percent.  Costs  associated  with  improving  the  RE  processes 
included  the  work  involved  in  changing  processes,  training  people  on  the  new  processes, 
doing  the  internal  ABB  CMMI  Class  B  Appraisal,  and  involving  the  corporate  process 
improvement  group.  The  only  benefit  included  in  the  ROI  calculation  is  the  savings  that 
resulted  from  the  30  percent  reduction  in  COPQ.  Amortizing  the  benefits  over  only  one  year, 
the  unit  achieved  a  2:1  return  on  investment  (ROI). 

Including  the  reduced  costs  of  developing  and  maintaining  the  product  and  amortizing  over  a 
longer  period  of  time  would  yield  a  less  conservative  ROI. 
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4.3  Hitachi  Software  Engineering 

written  with  Mutsumi  Komuro 


Background 

Organization  Background 

Founded  in  1971,  Hitachi  Software  Engineering  is  a  global  supplier  of  IT  services  and 
employs  over  5,000  people  worldwide.  With  sales  of  over  $2.1  billion  in  FY  2003,  Hitachi 
Software  provides  products,  professional  services,  consulting,  and  support. 

Hitachi  Software  Engineering  is  one  of  the  largest  software  companies  in  Japan  and  has  four 
major  groups  for  different  business  domains: 

•  Industrial  Systems  Group  (ISG) 

•  Financial  Systems  Group 

•  Public  and  Social  Systems  Group 

•  Software  Development  Group 

The  improvement  activities  and  performance  results  described  in  this  case  description  are 
mainly  from  ISG  The  customers  of  ISG  make  industrial  products  and  include  automobile 
makers,  electronic  device  makers,  and  filmmakers.  Typical  development  products  and 
services  of  ISG  include  customer  relationship  management  systems,  enterprise  resource 
planning  (ERP)-based  systems,  and  software  for  embedded  systems  and  information 
appliances. 

The  ERP -based  systems  are  supported  by  SAP  R3,  a  software  package  created  by  SAP  Labs 
in  Germany.  SAP  R3  integrates  and  automates  many  of  the  business  practices  associated  with 
operations,  production,  and  distribution  involved  in  manufacturing  products  or  services. 
Hitachi  Software  Engineering  also  provides  consulting  and  customizing  services  for  other 
users  of  SAP  R3. 

The  size  of  products  produced  at  ISG  varies  from  small  to  medium,  when  compared  to  the 
other  major  groups  in  Hitachi  Software  Engineering,  but  the  complexity  and  criticality  are 
relatively  higher  because  some  products  are  used  as  the  infrastructure  for  developing 
embedded  products  and  information  appliances. 

Process  Improvement  History 

Hitachi  Software  Engineering  had  little  prior  experience  with  model-based  process 
improvement  initiatives  or  standards.  Only  the  Software  Development  Group  has  experience 
with  ISO  9000,  and  none  of  the  four  groups  has  had  experience  with  Six  Sigma  approaches. 
The  company  never  used  the  SW-CMM  to  guide  its  process  improvement  activities. 
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Instead  of  using  common  process  improvement  standards,  Hitachi  Software  has  been 
conducting  two  kinds  of  periodic  internal  “movements”:  quality  improvement  movements 
and  productivity  enhancement  movements.  Each  movement  has  a  duration  of  two  to  three 
years,  and  they  have  been  conducted  since  the  establishment  of  Hitachi  Software 
Engineering.  Currently,  the  10th  quality  improvement  movement  and  the  11th  productivity 
enhancement  movement  are  being  conducted  company  wide.  ISG  also  conducted  a  separate 
code  inspection  movement  that  was  sponsored  by  the  ISG  general  manager  to  enhance  the 
quality  of  ISG  products. 

A  company-wide  process  improvement  initiative  based  on  CMMI  was  launched  in  2001.  The 
initiative  started  because  of  competition  Hitachi  Software  faced  in  its  industry.  The 
organization  wanted  to  stay  ahead  of  the  pack,  and  at  that  time  the  Japanese  government  was 
considering  requiring  a  Japanese  version  of  CMMI  for  all  suppliers. 

All  four  of  the  major  business  groups  at  Hitachi  Software  Engineering  achieved  CMMI 
maturity  level  3  in  2002.  The  Public  and  Social  Systems  Group  achieved  level  3  in  January, 
followed  by  the  Financial  Systems  Group  in  June  and  the  Software  Development  Group  in 
July.  The  Industrial  Systems  Group  achieved  maturity  level  3  in  October.  In  addition  to 
providing  a  thorough  understanding  of  the  strengths  and  weaknesses  of  the  groups’  then- 
current  practices,  preparing  for  the  maturity  level  3  appraisals  and  their  associated  team 
training  provided  an  important  focus  on  process  issues  to  help  kick  off  the  CMMI-based 
improvement  initiative  [Komuro  03]. 

ISG  has  approximately  800  employees  and  is  very  eager  to  train  them  on  its  standard 
processes,  which  are  based  heavily  on  CMMI.  The  SEPGsm  group  at  ISG  typically  consists  of 
4  to  5  full-time  members  and  20  to  30  part-time  members  who  also  serve  on  several 
supporting  working  groups. 


CMMI-Based  Process  Improvement 

SCAMPI  appraisals  conducted  prior  to  the  start  of  its  process  improvement  initiative  based 
on  CMMI  showed  the  need  for  improvement  at  Hitachi  Software  Engineering.  In  particular, 
the  appraisals  showed  that  the  organization  needed  to  be  less  product  centric  and  focus  on 
process  as  well  as  product.  As  stated  before,  ISG  achieved  CMMI  maturity  level  3  in  October 
2002.  A  focus  on  improving  the  organization’s  peer  review  processes  helped  ISG  achieve 
maturity  level  4  in  February  2004  and  maturity  level  5  in  October  2004. 

ISG  began  by  evaluating  its  existing  practices  in  order  to  incorporate  the  best  ones  into  its 
new  and  revised  peer  review  processes.  Relationships  of  improvement  activities  to  business 
goals  were  clearly  stated,  and  project  members  were  involved  in  the  improvement  of  their 
own  processes. 

A  fair  amount  of  performance  data  were  available  from  the  ISG  code  inspection  movement 
by  the  time  that  the  organization  achieved  CMMI  maturity  level  3.  The  performance  data 
were  analyzed  statistically  as  ISG  moved  toward  CMMI  maturity  levels  4  and  5.  There 
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initially  were  some  flaws  in  the  measurement  process  (e.g.,  some  data  seemed  to  be 
incorrectly  recorded  and  some  key  measures  were  missing);  however,  the  early  results 
encouraged  ISG  to  continue  improving  its  code  inspection  practices.  The  following 
improvements  were  implemented: 

•  The  measurement  process  and  definitions  of  measures  were  made  clearer. 

•  Peer  review  performance  objectives  were  related  explicitly  to  business  goals. 

•  Existing  best  practices  found  during  the  analysis  were  recorded. 

•  A  tool  to  plot  control  charts  was  developed  using  Excel  macros. 

•  Training  courses  were  conducted  for  the  first  four  activities. 

•  An  ISG  service  was  provided  to  support  the  projects  in  the  analysis  of  their  peer  review 
data. 

•  ISG  provided  a  project  support  service  to  analyze  source  code,  using  Hitachi  Software 
Engineering’s  Code  Director  tool  to  identify  static  semantic  errors  and  failure  to  follow 
coding  standards. 

Prior  to  the  CMMI  initiative  at  ISG,  quality  assurance  (QA)  activities  consisted  of 
independent  verification  of  various  work  products.  The  projects  first  verified  their  own  work 
products,  including  various  specification  documents  and  source  code.  The  same  documents 
and  programs  then  were  verified  by  QA  departments  that  report  directly  to  the  ISG  board, 
independently  from  the  projects,  using  their  own  checklists  and  test  sets. 

While  QA  representatives  usually  participated  in  progress  meetings  with  the  projects,  they 
did  not  check  the  projects’  ongoing  process  activities  often.  The  main  focus  of  the  QA 
activities  was  to  check  the  work  products  rather  than  the  processes.  The  CMMI  Process  and 
Product  Quality  Assurance  process  area  had  a  positive  effect  on  product  quality  because  it 
helped  guide  ISG  to  conduct  quality  assurance  on  process  as  well  as  product  characteristics. 

Other  important  process  improvements  advocated  by  CMMI  included  the  following: 

•  An  ISG  measurement  system  with  an  emphasis  on  testing  was  in  place  long  before  the 
organization  began  using  CMMI;  however,  a  number  of  flaws  were  recognized  based  on 
guidance  found  in  the  Measurement  and  Analysis  process  area.  In  particular,  the  process 
area  encouraged  the  organization  to  explicitly  relate  measurement  to  its  high  priority 
business  goals. 

•  While  ISG  previously  did  have  informal  processes  to  aid  in  the  making  of  major 
decisions,  the  organization  used  the  Decision  Analysis  and  Resolution  process  area  to 
formalize  those  processes,  especially  for  comparing  alternatives. 

•  As  noted  above,  the  organization  also  relied  heavily  on  activities  advocated  by  CMMI  for 
Quantitative  Project  Management,  Causal  Analysis  and  Resolution,  and  Organizational 
Innovation  and  Deployment  to  improve  its  peer  review  processes. 
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Testing  had  been  emphasized  previously,  and  it  was  not  an  improvement  priority  for  CMMI- 
based  improvement  at  ISG;  however,  reducing  testing  costs  was  a  major  emphasis  for  the 
organization’s  higher  maturity  effort. 

Performance  Results 

The  performance  results  presented  here  focus  on  those  due  to  improvements  in  ISG  processes 
for  peer  review  and  quality  assurance.  The  results  are  drawn  from  pilot  studies  and  other 
statistical  analyses  conducted  prior  to  wider  scale  deployment  throughout  the  organization. 
The  peer  review  results  are  based  largely  on  hypothesis  testing  using  regression  analysis 
methods  as  well  as  the  statistical  process  control  techniques  used  for  quantitative  project 
management  at  ISG.  The  quality  assurance  results  are  based  on  pilot  testing. 

The  organization’s  measurement  definitions  and  procedures  have  changed  as  its  measurement 
capabilities  matured;  therefore,  ISG  has  not  calculated  overall  performance  improvements 
resulting  from  adopting  CMMI  or  achieving  higher  maturity  levels. 

Early  in  its  CMMI  improvement  initiative,  ISG  observed  a  strong  negative  correlation 
(r2=.72)  between  defect  detection  prior  to  unit  testing  and  total  defect  density  throughout  the 
product  life  cycle.  Projects  that  found  defects  earlier  also  had  substantially  fewer  total 
defects.  Moreover,  the  effort  and  cost  to  find  and  fix  defects  captured  at  unit  test  proved  to  be 
2.28  times  more  costly  than  those  found  during  peer  review.  This  was  true  even  though  the 
early  peer  reviews  were  less  efficient  than  those  based  on  subsequent  process  improvement. 
Improvements  to  peer  reviews  at  ISG  have  improved  staff  productivity  and  reduced  costs 
while  improving  product  quality. 

Early  in  their  implementation  of  quantitative  project  management  activities,  ISG  engineers 
also  noticed  that  Fagan-style  inspections  used  in  their  exemplary  projects  seemed  to  work 
better  than  the  less  formal  peer  review  walkthroughs  that  were  usually  conducted  elsewhere 
in  ISG.  Relatively  little  data  yet  existed  for  the  Fagan-style  inspections,  and  the  organization 
had  not  yet  established  separate  baselines;  however,  regression  and  other  statistical  analyses 
were  able  to  detect  differences  ranging  from  almost  4.5  to  over  10  times  more  defects  found 
with  the  inspections.  More  defects  were  found  earlier  with  Fagan-style  inspections  when 
correcting  them  was  substantially  less  expensive. 

Possibly  because  there  was  a  learning  curve  associated  with  the  first  few  instantiations  of  the 
Fagan-style  inspections,  ISG  was  not  able  to  confirm  from  its  early  results  that  its  Fagan- 
style  inspections  were  more  efficient  than  its  walkthroughs.  While  the  inspections  clearly 
found  more  defects  than  the  walkthroughs,  more  effort  was  spent  on  preparing  for  the 
inspections.  Still,  the  ISG  measure  of  peer  review  efficiency  was  based  on  the  number  of 
defects  identified  in  a  peer  review  divided  by  the  effort  spent  on  the  peer  review  only.  The 
cost  of  correcting  the  defects  is  substantially  more  expensive  when  they  are  found  later  in  the 
product  life  cycle. 
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Looking  more  closely  at  special  causes  of  variation,  ISG  identified  a  series  of  peer  review 
practices  that  it  then  incorporated  in  other  pilot  projects.  These  included  clarifying  in  advance 
what  aspect  of  the  work  product  should  be  checked,  limiting  the  duration  of  the  reviews,  and 
limiting  the  number  of  participants  in  the  reviews. 

Subsequent  piloting  also  found  that  more  defects  escaped  detection  during  peer  review  when 
ISG  tried  to  cover  too  large  a  product  in  a  single  review;  therefore,  the  organization 
established  an  upper  limit  for  review  size.  As  seen  in  Figure  22,  though,  use  of  the  Hitachi 
Software  Engineering  Code  Director  static  analysis  tool  increased  the  upper  limit  of 
manageable  review  size  by  30  percent.  Overall,  the  return  on  investment  for  the  Fagan-type 
inspections  was  3.9:1  as  opposed  to  postponing  verification  until  unit  test.  The  comparable 
return  for  the  less  formal  walkthroughs  was  2.3:1. 


Difference  of  defect  densities  is  significant  here 
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Figure  22:  Peer  Review  With  and  Without  Tool  Aid 

Prior  to  deployment  throughout  the  organization,  ISG  conducted  a  series  of  pilot  studies  of 
the  effects  of  including  process  along  with  product  in  its  quality  assurance  activities.  Data 
were  gathered  from  16  pilot  projects,  one  or  two  of  which  were  selected  from  each  ISG 
department.  Defect  density  in  those  projects  was  15  percent  lower  than  the  ISG  baseline.  In 
addition  to  the  improvement  in  product  quality,  the  organization  calculated  a  return  on 
investment  (ROI)  of  1.7:1  due  largely  to  reduction  in  the  cost  of  subsequent  verification  and 
validation  activities.  Separate  analyses  conducted  by  the  QA  departments  reported  ROI  ratios 
ranging  from  6.3:1  to  9.8:1  when  the  costs  of  correcting  field  errors  and  other  management 
and  engineering  problems  were  factored  into  the  ROI  ratios. 
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4.4  Motorola  Global  Software  Group 

The  Motorola  Global  Software  Group  (GSG)  develops  and  maintains  custom  software  for 
many  Motorola  products  and  services.  GSG  provides  technical  leadership  and  expertise  to 
deliver  advanced  solutions  for  software  that  must  be  used  seamlessly  throughout  the 
corporation  and  by  its  customers  worldwide.  With  a  flexible  pool  of  over  6000  engineers,  a 
total  of  14  GSG  Software  Design  Centers  are  distributed  throughout  the  world.  This  section 
contains  brief  case  descriptions  detailing  the  performance  results  achieved  through  CMMI- 
based  process  improvement  by  the  Centers  in  China,  India,  and  Russia. 

4.4.1  Motorola  GSG  China  Center 

written  with  Angel  (Liu  Qi)  Liu 


Background 

The  Organization  and  Its  Products 

Like  all  Motorola  Global  Software  Group  Software  Design  Centers,  the  primary  business  of 
the  GSG  China  Center  is  to  provide  software  development  services  and  solutions  to  other 
Motorola  business  units.  GSG  China’s  products  include  various  embedded  systems  in 
cellular,  network  system,  and  other  telecommunication  devices.  GSG  China  was  established 
in  1993.  There  are  three  sites  in  China,  in  Beijing,  Chengdu,  and  Nanjing,  and  GSG  China 
currently  employs  a  staff  of  more  than  1200  individuals. 

Work  throughout  the  Global  Software  Group  is  organized  into  several  business  divisions  that 
align  with  Motorola’s  major  corporate  business  units.  The  China  Center  does  most  of  its 
business  for  three  GSG  divisions: 

•  The  Embedded  Products  and  Systems  Division  provides  innovative  and  rapid  software 
solutions  for  Motorola’s  consumer  products. 

•  The  Infrastructure  Solutions  Division  provides  software  products  and  systems  solutions 
for  the  Motorola  network  telecommunication  business  and  its  customers  worldwide. 

•  The  Integrated  Communication  Solution  Division  provides  software  and  systems 
solutions  for  the  professional  two-way  radio  businesses. 

GSG  China  provides  end-to-end  solutions  in  many  areas,  including  mobile  terminal  software 
development;  wireless;  embedded  system  software;  professional  two-way  radio  software 
development  and  testing;  network  management;  and  automotive  telecommunication  and 
system  software  development.  GSG  China  has  led  many  regional  and  global  projects,  and  its 
software  products  and  services  are  widely  applied  in  Motorola  products  across  a  variety  of 
platforms.  The  Center’s  products  vary  in  size  from  18,000  non-comment  source  statements 
(KNCSS)  to  381  KNCSS.  The  China  Center’s  projects  include  designing,  developing,  testing, 
porting,  reverse  engineering,  and  performing  other  services. 
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In  addition  to  its  heritage  in  providing  end-to-end  solutions  in  the  various  fields  just 
mentioned,  GSG  China  now  is  working  with  Motorola  Software  Technology  Groups, 
particularly  in  the  areas  of  software  engineering  tools,  intelligent  user  interfaces,  open  source 
technology  and  embedded  systems,  and  seamless  mobility  enablers. 

Process  Improvement  History 

GSG  China  was  the  first  software  organization  in  China  to  use  the  SW-CMM  as  a  basis  for 
their  process  improvement  program.  In  2000,  this  organization  was  the  first  in  China  to  be 
appraised  at  maturity  level  5.  Process  engineers  in  the  China  Center  began  focusing  their 
attention  on  CMMI  in  2002.  By  the  end  of  2003,  the  China  Center  had  begun  its  formal 
transition  to  CMMI,  and  it  was  appraised  at  maturity  level  5  in  September  of  2005. 

Like  all  Motorola  GSG  Centers,  the  one  in  China  uses  Digital  Six  Sigma  in  concert  with  its 
CMMI-based  improvement  efforts.  Digital  Six  Sigma  is  a  Motorola  deployment  that  stresses 
e-leaming  and  automation  to  ensure  the  institutionalization  of  process  improvements. 

Some  GSG  China  operations  also  use  TL9000  audits  in  concert  with  CMMI.  Created  to  meet 
the  quality  requirements  of  the  worldwide  telecommunications  industry,  the  TL  9000  Quality 
Management  System  provides  additional  insight  into  telecommunications  processes  for 
resource  management,  disaster  recovery,  device  control,  and  preservation  of  product  [QuEST 
06]. 

CMMI-Based  Process  Improvement 

For  GSG  China,  the  transition  to  CMMI  was  part  of  a  natural  progression  of  continuous 
CMM-based  improvement.  A  CMMI  taskforce  was  established  in  November  of  2001.  The 
Center’s  process  engineers  began  training  and  performed  a  gap  analysis  in  April  of  2002. 

By  early  2003,  senior  management  authorized  the  process  engineers  to  begin  piloting  and 
start  training  on  what  it  considered  to  be  critical  new  practices  based  on  the  CMMI  Project 
Management  and  Engineering  process  areas.  Particular  attention  was  paid  to  the  CMMI 
upgrades  to  Project  Planning,  Project  Monitoring  and  Control,  Requirements  Management, 
and  Verification.  Full  scale  planning  and  modification  of  the  Center’s  Organizational 
Standard  Process  began  in  December  of  2003.  Changes  to  the  software  production  process 
then  were  piloted  and  deployed  throughout  the  Center’s  operations. 

Major  process  improvements  achieved  through  GSG  China’s  transition  to  CMMI  include  the 
following: 

•  The  GSG  Process  Hierarchy  Structure  was  adapted  for  use  by  the  China  Center  to 
accommodate  process  changes  due  to  CMMI.  It  is  organized  by  process  to  assist  the 
projects  on  process  model  selection,  and  it  provides  guidance  and  flexibility  for  projects 
to  select  and  tailor  their  processes  at  a  detailed  subprocess  level.  The  process  asset  library 
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can  be  accessed  by  several  criteria,  including  CMMI  process  areas,  phase,  process 
templates,  checklists,  and  other  types  of  documents. 

•  The  Center’s  engineering  processes,  particularly  for  requirements  development,  unit  test, 
and  validation  now  provide  more  practical  guidelines,  templates,  techniques,  and  tools. 

•  The  increased  emphasis  on  CMMI  in  organizational  processes  led  GSG  China  to  deploy 
a  COTS  enterprise  project  management  system  as  a  common  project  management 
platform.  Customized  by  Motorola  corporate,  it  supports  GSG  China’s  multi-site  team  to 
work  in  an  integrated  manner  on  resource  management,  time  tracking,  project  planning, 
milestone  tracking,  risk  management,  and  related  activities.  All  project  status  and 
performance  can  be  planned,  tracked,  and  measured  in  a  quantitative  manner,  and  it 
provides  a  global  view  for  project  and  infrastructure  management  at  all  levels. 

•  The  Center’s  quantitative  project  management  processes  were  modified  to  focus  more  on 
critical  processes  aligned  with  business  objectives,  particularly  in  the  areas  of 
requirements  management  and  verification. 

•  Causal  Analysis  and  Resolution  (CAR)  and  Organizational  Innovation  and  Deployment 
(OID)-based  processes  were  used  to  improve  verification  process  effectiveness  without 
compromising  quality  in  specific  classes  of  cases.  The  formal  process  rules  the  Center 
previously  used  were  less  effective  for  smaller  documents  and  project  teams.  Mini¬ 
reviews  and  mini-inspections  were  introduced;  effort  and  cost  were  reduced  without 
affecting  defect  density.  The  results  were  used  to  establish  new  baselines  for  planning 
and  quantitative  project  management. 

•  The  Center’s  requirements  development  and  management  process  was  improved  based 
on  guidance  in  CMMI  that  did  not  exist  in  the  SW-CMM.  Late  additions  and  changes  to 
requirements  are  measured.  The  project  level  results  then  are  rolled  up  to  the 
organizational  level  for  senior  management  insight  and  to  set  control  limits  for  future  use 
in  quantitative  process  management  of  requirements  volatility.  The  measured  and 
estimated  impact  of  the  requirements  changes  also  led  to  new  project  processes  for 
replanning,  estimation,  and  a  factual  basis  for  negotiation  with  customers. 

•  New  GSG  China  practices  based  on  the  CMMI  Decision  Analysis  and  Resolution  (DAR) 
process  area  were  used  to  improve  the  Center’s  managerial  and  engineering  processes, 
particularly  those  that  map  to  the  Technical  Solution,  Configuration  Management,  and 
OID  process  areas.  These  DAR-based  practices  have  been  applied  to  improve  decision 
making  effectiveness  at  both  the  organization  and  project  level. 

As  noted  earlier,  the  GSG  China  CMMI  transition  project  began  in  December  of  2003.  It 
continued  for  22  months  through  September  of  2005.  The  total  effort  spent  was 
approximately  17.6  staff  years,  which  is  about  1.1  percent  of  the  Center’s  total  engineering 
effort.  Most  of  the  effort,  60  percent,  was  spent  on  training  for  deployment.  About  20  percent 
was  used  on  process  redesign,  and  14  percent  was  devoted  to  appraisal  activities.  More  than 
92  percent  of  the  employees  received  classroom  training  on  the  new  GSG  China  software 
production  process. 
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During  the  transition,  the  areas  in  need  of  improvement  were  identified  by  analyzing  the  gap 
between  CMMI  and  the  Center’s  then  current  processes.  Priorities  were  set  based  on  the 
impact  and  urgency  of  the  improvements.  For  major  improvements,  the  Center  followed  an 
OID-based  process  including  evaluation,  pilot,  and  deployment.  During  the  piloting  phase,  3 1 
process  assets  were  selected  to  be  piloted  in  14  projects  that  covered  all  of  the  Center’s  major 
areas  of  business.  Since  this  transition,  GSG  China  has  continued  to  renew  its  process 
architecture  and  assets.  At  the  time  of  this  writing,  182  new  and  revised  process  assets  have 
been  deployed. 

Performance  Results 

Cost  of  quality,  estimation  accuracy,  and  product  quality  all  have  improved  markedly  as  the 
Motorola  Global  Software  Group  China  Center  upgraded  to  CMMI.  The  Center’s  customer 
satisfaction  ratings,  which  already  were  very  high,  continued  to  improve.  The  customers 
continued  to  report  that  the  Center  exceeded  their  expectations. 

All  of  the  measurement  definitions  and  quantitative  results  reported  here  are  maintained  in  a 
Motorola  GSG  central  data  warehouse.  The  results  are  based  on  170  projects  of  the  China 
Center  from  2000  through  2005.  In  addition  to  the  results  presented  here,  GSG  China 
regularly  manages  earned  value  quantitatively. 

As  a  result  of  improved  verification  processes,  particularly  product  peer  review  and  software 
test  processes,  GSG  China  was  able  to  reduce  its  overall  cost  of  quality  by  over  one  third, 

36.5  percent,  from  its  pre-CMMI  baseline  (see  Figure  23).  The  cost  of  poor  quality  remained 
under  control  at  less  than  5  percent  during  the  same  period. 


Cost  Of  Quality 


Figure  23:  Cost  of  Quality 
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As  seen  in  Figure  24,  the  China  Center  was  able  to  improve  its  product  quality  at  the  same 
time  that  it  reduced  its  cost  of  quality.  Again,  the  improvement  in  performance  was  largely 
due  to  improved  verification  processes,  which  had  been  adapted  through  application  of  the 
Center’s  improved  casual  analysis  and  resolution  procedures.  Fewer  defects  were  inserted, 
and  the  Center  was  able  to  reduce  its  software  faults  per  thousand  lines  of  code  found  prior  to 
release  by  40.2  percent  from  its  pre-CMMI  capability. 


Faults  Per  Thousands  Line  Of  Code 


Figure  24:  In  Process  Faults  per  Thousand  Lines  of  Code 

Particularly  through  their  improved  project  planning,  estimation,  and  tracking  practices,  GSG 
China  was  able  to  improve  its  initial  effort  estimation  accuracy  by  almost  a  third,  31.4 
percent,  as  the  organization  moved  to  CMMI  maturity  level  5  from  its  SW-CMM  maturity 
level  5  baseline.  During  the  same  time  period,  the  accuracy  of  estimated  schedule  duration 
improved  by  well  over  three  quarters,  84.2  percent,  over  its  performance  at  SW-CMM 
maturity  level  5.  The  percentages  shown  in  Figure  25  are  based  on  the  absolute  values  of  the 
differences  between  the  respective  actual  and  estimated  values  divided  by  the  original 
estimates  for  each  time  period. 
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Figure  25:  First  Estimation  Accuracy 

With  a  sustained  on-time  delivery  rate  of  100  percent,  the  GSG  China  Center  has  consistently 
received  high  scores  on  its  customer  satisfaction  surveys,  which  are  conducted  at  the 
conclusion  of  every  project  (Figure  26).  The  surveys  follow  standard  GSG  practice,  with 
scores  of  8  out  of  10  being  considered  “very  good”  and  9  representing  “excellent” 
performance.  Performance  goals  are  updated  every  year.  The  China  Center’s  goal  for  2005 
was  8.86;  it  achieved  an  average  score  over  all  of  its  surveys  of  9.03.  Since  the  transition  to 
CMMI,  the  surveys  show  that  GSG  China  has  continued  to  exceed  its  customers’ 
expectations.  It  has  continued  to  achieve  high  customer  satisfaction  levels,  as  measured  by 
the  customers’  answers  to  explicit  questions  about  the  Center’s  ability  to  deliver  cost 
effective,  high  quality  products  on  time  by  providing  excellent  technical  solutions  and  project 
execution. 
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Overall  Customer  Satisfaction 
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Figure  26:  Overall  Customer  Satisfaction 

The  Motorola  GSG  China  Center  has  grown  in  both  staff  size  and  business  volume  since 
beginning  its  journey  to  CMMI  maturity  level  5.  By  the  end  of  2005,  the  China  center’s  staff 
was  almost  double  what  it  had  been  in  2003.  Its  business  volume  and  revenue  increased 
substantially  by  55  percent  during  the  same  time  period.  The  Center  believes  that  its 
continuous  improvement  of  an  already  highly  mature  process  has  been  an  important 
competitive  advantage. 

4.4.2  Motorola  GSG  India  Center 

written  with  Balaji.R 


Background 

The  Organization  and  Its  Products 

Motorola  India  Electronics  Ltd  (MIEL)  was  founded  in  1991  and  had  60  employees  in  the 
first  year.  The  organization  had  grown  to  800  employees  prior  to  its  transition  to  CMMI  in 
March  of  2003.  By  February  of  2006,  the  organization  had  tripled  in  size  to  2400  employees. 
The  organization  attributes  its  ability  to  manage  its  process  culture  while  accommodating 
such  rapid  growth  to  the  additional  guidance  provided  by  CMMI  best  practices. 

MIEL  develops  and  maintains  telecom  software  for  Motorola  cell  phone  products,  wireless 
networks,  broad  band  cable  modems,  and  automotive  and  home  solutions.  Motorola  Global 
Software  Group's  work  is  represented  and  organized  by  several  business-centric  divisions. 

The  Embedded  Products  and  Systems  Division  (EPSD)  is  the  largest  division  in  GSG  India. 

It  provides  innovative  and  rapid  software  solutions  for  Motorola’s  consumer  products.  The 
MIEL  Infrastructure  Solutions  Division  (ISD)  provides  customer  software,  software  products, 
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and  systems  solutions  to  the  Motorola  Networks  telecommunications  business  and  its 
customers  worldwide. 

MIEL  provides  services  from  requirements  to  field  test  and  shipment  of  the  products  to  the 
market.  The  products  continue  to  grow  in  size  and  increasingly  require  additional  innovative 
and  difficult  features.  Since  2002,  typical  product  size  has  grown  from  130  KLOC  to  2218 
KLOC,  mostly  in  C  and  C++  code. 

Process  Improvement  History 

Prior  to  embarking  on  its  move  to  CMMI,  MIEL  had  been  appraised  at  SW-CMM  maturity 
level  5.  The  organization  made  the  decision  to  upgrade  to  CMMI  in  March  of  2003.  Other 
than  mapping  and  indexing  its  process  assets  to  the  new  model,  MIEL  had  relatively  little  to 
accomplish  to  meet  the  criteria  for  appraisal  at  CMMI  maturity  level  5  in  May  of  2003.  In 
addition  to  its  prior  experience  with  the  SW-CMM,  MIEL  is  compliant  with  ISO  9000  and 
TL9000;  the  latter  of  which  is  specific  to  telecom  standards  [QuEST  06].  Like  all  Motorola 
GSG  Centers,  MIEL  also  has  a  prior  and  continuing  improvement  history  based  on  Digital 
Six  Sigma  methods,  applied  in  particular  to  telecom  call  performance. 


CMMI-Based  Process  Improvement 

Several  new  and  changed  processes  were  implemented  as  part  of  the  transition  to  CMMI  at 
Motorola  India  Electronics  Ltd.  While  many  maturity  level  4  and  5  processes  had  been 
initiated  during  the  SW-CMM  period,  substantial  changes  in  MIEL  processes  were  deployed 
throughout  the  organization  due  to  continuing  improvements  based  on  CMMI.  The  changes 
from  which  the  performance  results  presented  here  originated  took  place  from  2003  through 
2005. 

Modeling  and  simulation  methods  based  on  the  Motorola  Digital  Six  Sigma  effort  were  used 
to  predict  tangible  performance  improvements  in  cost  of  quality  and  poor  quality.  These  were 
followed  by  piloting,  prior  to  full  deployment,  using  standard  processes  that  had  been 
initiated  during  the  SW-CMM  period.  MIEL  used  the  continuous  representation  of  CMMI  to 
guide  their  improvement  efforts.  All  of  the  following  changes  to  MIEL  processes  were  based 
explicitly  on  guidance  that  was  not  present  in  the  SW-CMM.  Moreover,  the  increased  formal 
discipline  of  many  of  these  new  processes  facilitated  MIEL’s  integration  with  automated  tool 
support  and  related  process  technologies. 

•  An  enhanced  inspection  process  was  introduced  and  institutionalized  using  causal 
analysis  and  resolution  procedures  to  improve  the  previous  inspection  process.  Self 
reviews  now  are  conducted  prior  to  formal  inspections.  Inspection  moderators,  who  are 
subject  matter  experts,  prescreen  the  self  inspection  results  before  calling  the  formal 
inspection  meetings.  The  moderators  now  can  send  the  self  inspections  back  for  rework, 
saving  a  great  deal  of  staff  time  and  effort  that  otherwise  would  be  spent  on  preparation 
and  conduct  of  the  formal  reviews. 
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•  Inspection  guidelines  are  modified  as  necessary  to  keep  preparation  rates  and  inspection 
time  within  baseline  limits.  Additional  root  cause  analysis  is  done  regularly  to  investigate 
classes  of  errors  found  later  in  the  product  life  cycle  after  the  formal  inspections.  Causes 
of  injection  for  such  defects  that  are  not  found  until  later  have  been  identified,  and  rules 
have  been  established  to  capture  them  earlier.  As  a  result  of  following  CMMI,  MIEL  also 
now  inspects  all  design  documents,  planning  documents,  and  other  major  work  products 
in  addition  to  100  percent  of  its  code. 

•  Automation  of  control  chart  applications  used  on  inspection  data  has  improved  the 
effectiveness  of  both  design  and  code  phase  containment.  The  regular  use  of  a 
commercial  off-the-shelf  (COTS)  verification  product  has  been  institutionalized  to 
reduce  code  defects  creeping  into  the  next  phase;  Klocwork  is  a  static  code  analyzer  that 
detects  memory  leakage,  array  indices  that  are  out  of  bounds,  and  related  errors 
[Klocwork  06]. 

•  Formal  Decision  Analysis  and  Resolution  (DAR)  practices  have  been  instituted  in  the 
engineering  phases  to  improve  various  product  quality  attributes.  In  particular,  a  rigorous 
DAR  process  now  is  followed  during  the  design  phase,  which  rates  alternatives  with 
specified  criteria. 

•  Major  changes  in  audit  methodology  have  been  instituted  for  both  process  and  product 
quality  assurance.  Different  types  of  audits,  both  focused  and  continuous,  now  use  in- 
house  tools  that  explicitly  identify  non-conformances  in  real  time  with  the  direct 
involvement  of  project  managers.  All  corrective  actions  are  established  in  real  time,  and 
they  are  monitored  and  formally  tracked  to  closure.  MIEL’s  audit  process  previously  used 
checklists  that  were  covered  subjectively  as  time  permitted;  now,  all  checkpoint  questions 
are  covered  regularly. 

•  A  COTS  tool  for  enterprise  project  management  is  now  being  used  to  improve  MIEL 
scheduling  and  tracking  of  process  compliance  [Primavera  06a]. 

•  Improvements  in  MIEL  validation  processes  have  brought  them  more  business  in  system 
integration  and  testing.  The  use  of  test  automation  tools  such  as  their  in-house-developed 
Simulated  Field  Environment  has  helped  MIEL  reduce  the  number  of  defects  found  in  the 
field.  For  similar  reasons,  MIEL  uses  a  COTS  product,  Smoke  Suite,  to  automatically  run 
test  cases.  MOTO_OATS,  an  in-house  Web-based  tool,  is  used  for  analyzing  orthogonal 
pairs  of  test  cases  to  handle  pair-wise  combinations  of  requirements.  This  has  improved 
the  writing  of  test  cases  to  handle  such  combinations.  As  seen  in  the  Performance  Results 
section  of  this  case  description,  these  improvements  have  resulted  in  more  effective 
defect  capturing  during  testing  and  less  staff  effort  devoted  to  test  design  and  execution. 

•  The  emphasis  in  CMMI  on  bi-directional  traceability  led  MIEL  to  upgrade  its 
requirements  management  process,  supported  by  tools  such  as  DOORS  [Telelogic  06] 
and  the  Rational  EmbeddedPlus  DoorKeeper  [IBM  06]. 
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Performance  Results 


Overall,  the  implementation  of  CMMI  at  Motorola  India  Electronics  Ltd  has  improved 
management  commitment  for  continuous,  disciplined  process  improvement.  The  performance 
results  reported  here  have  contributed  to  that  commitment  and  to  the  organization’s  ability  to 
triple  its  size  in  the  three  years  since  its  movement  to  CMMI.  In  particular,  management  saw 
additional  value  in  the  emphasis  in  CMMI  on  continuous  capability  improvement  and 
organizational  level  monitoring. 

The  Measures 

MIEL  regularly  collects  and  analyzes  data  for  several  performance  measures.  All  of  the  data 
are  collected  and  managed  using  Web-based  tools  that  were  developed  in-house  and  are  well 
integrated  with  the  organization’s  standard  processes.  The  following  measures  that  MIEL 
uses  are  categorized  by  their  mappings  to  CMMI  process  areas. 

•  Requirements  Management:  Requirements  phase  containment  effectiveness  (PCE)  and 
requirements  volatility 

•  Technical  Solution:  Design  phase  containment  effectiveness,  code  phase  containment 
effectiveness,  and  defect  density 

•  Verification:  Review  fault  density,  inspection  rate  and  inspection  preparation  rate  (both 
counted  in  lines  of  code  or  pages  per  hour),  review  cost  of  quality,  and  review  cost  of 
poor  quality 

•  Validation:  Test  cost  of  quality,  test  cost  of  poor  quality,  in-process  faults  (major  errors 
and  defects  divided  by  product  size),  post  release  defects,  and  phase  screening 
effectiveness  (similar  to  phase  containment  effectiveness  but  applied  to  test  phases) 

•  Project  Planning  and  Project  Monitoring  and  Control:  On  time  delivery,  effort,  size 
and  schedule  estimation  accuracies,  productivity  (lines  of  code  divided  by  staff  days), 
and  earned  value  analysis 

All  phase  containment  effectiveness  (PCE)  measures  are  expressed  as  percentages  of  defects 
identified  in  the  phase  where  they  were  injected.  Defect  density,  review  fault  density  and  test 
fault  density  measures  are  numbers  of  faults  divided  by  product  size.  Post-release  defects  are 
simple  counts.  Cost  of  quality  is  measured  as  percentage  of  staff  effort  spent  on  ensuring 
product  quality.  Cost  of  poor  quality  is  measured  as  percentage  of  staff  effort  devoted  to 
rework.  In  addition,  MIEL  collects  process  compliance  measures  such  as  number  of  non- 
compliances  per  audit  and  number  of  audits  per  month. 

Project-level  analyses  are  performed  monthly  using  these  measures.  The  results  of  those 
analyses  are  used  as  needed  for  process  improvements  and  other  corrective  actions. 
Organization  level  quarterly  analyses  are  done  to  update  organizational  process  enactment 
and  performance  baselines. 
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The  following  points  deserve  emphasis  in  interpreting  the  results  that  follow: 

•  Other  things  being  equal,  the  baseline  year  for  comparison  of  the  effects  due  to  CMMI- 
based  improvement  would  be  2002,  at  which  time  the  organization  still  was  following 
SW-CMM  appraised  maturity  level  5  processes. 

•  Comparisons  between  2002  and  the  years  that  followed  are  difficult  though,  since  MIEL 
measurement  procedures  changed  considerably  after  2002.  Procedures  have  been 
instituted  to  improve  measurement  validity.  Moreover,  the  defect  and  related 
measurement  definitions  have  changed.  Prior  to  2003  MIEL  did  not  include  previously 
existing  defects  that  were  injected  prior  to  the  time  MIEL  took  over  the  work  on  pre¬ 
existing  software  products.  By  now,  most  of  those  defects  have  been  found,  so  including 
the  2002  results  in  the  figures  below  would  understate  the  improvement  in  the  CMMI 
period  that  followed. 

•  The  development  efforts  undertaken  since  2002  remain  essentially  similar  in  application 
domain;  however,  the  degree  of  difficulty  of  the  work  has  increased. 

•  MIEL  was  appraised  at  CMMI  Maturity  level  5  in  2003.  Particularly  notable  additional 
work  using  Digital  Six  Sigma  tools  and  techniques  was  initiated  in  concert  with  CMMI- 
based  improvement  during  the  same  year. 

•  Additional  implementation  of  MIEL  processes  that  map  to  the  Project  Planning,  Project 
Monitoring  and  Control,  Verification,  and  Validation  process  areas  were  instituted  in 
2004  via  their  enhanced  inspection  process,  EPMS,  and  Teamplay  [Primavera  06b]. 

•  Major  audit  process  improvements  took  place  in  2005. 

•  The  results  are  based  on  40  projects  each  year. 

The  Results 

Automation  of  its  audit  process  helped  MIEL  improve  its  ability  to  detect  non-compliances  in 
its  process  and  product  quality  assurance  efforts.  By  2005,  the  organization  was  able  to  detect 
412  non-compliances  in  50  audits  conducted  over  a  three-month  period.  Earlier  in  the  same 
year,  it  found  109  non-compliances  in  82  audits  that  were  conducted  in  the  same  time  frame. 
This  improvement  is  largely  due  to  the  major  changes  in  audit  methodology  that  MIEL 
undertook  based  on  guidance  from  the  CMMI  Process  and  Product  Quality  Assurance 
process  area. 

Figure  27  shows  that  the  organization’s  requirements  phase  containment  effectiveness  (PCE) 
was  sustained  at  90  percent  in  spite  of  the  enormous  organizational  growth  and  the  inclusion 
of  more  projects  that  focused  on  later  stage  life-cycle  products.  Similarly,  the  design  and  code 
PCE  improved  by  8  percent  and  12  percent  respectively,  even  when  expressed  conservatively 
as  fractions  of  100  percent  of  total  phase  containment  effectiveness.3  In  the  end,  the 

3  There  is  only  limited  room  left  for  improvement  over  the  2003  baseline  of  defects  that  had 

previously  escaped  detection.  The  same  differences  would  appear  greater  if  they  were  expressed 
as  percentages  of  the  possible  room  left  for  improvement. 
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organization  was  able  to  contain  more  faults  well  within  its  development  phases,  thereby 
reducing  rework  as  well  as  the  defects  that  otherwise  may  have  been  delivered  to  customers. 
Much  of  this  improvement  resulted  from  process  improvements  based  on  the  CMMI 
Verification  and  Organizational  Training  process  areas. 
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Figure  27:  Phase  Containment  Effectiveness 

Major  in-process  faults4  (IPF)  also  have  fallen  since  2003,  from  5.21  to  3.22  faults  per  KLOC 
(Figure  28).  Hence,  the  faults  delivered  to  customers  with  respect  to  the  product  size  have 
been  reduced  substantially.  Much  of  the  decrease  took  place  as  a  result  of  the 
institutionalization  of  improved  processes  based  on  the  CMMI  Causal  Analysis  and 
Resolution,  Organizational  Innovation  and  Deployment,  Technical  Solution,  and  Decision 
Analysis  and  Resolution  process  areas. 


Figure  28:  In-Process  Faults 


4  Not  all  projects  at  MIEL  cover  the  full  scope  of  the  organization’s  standard  software  process.  For 
example,  code  and  unit  test  may  be  the  final  phase.  Comparable  to  measures  elsewhere  of  defect 
density  prior  to  release,  in-process  faults  (IPF)  at  MIEL  include  all  of  the  faults  identified  by  the 
final  phase  of  each  project. 
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The  decline  in  in-process  faults  also  is  shown  in  Figure  28  expressed  as  faults  per  thousand 
assembly-equivalent  lines  of  code  (KAELOC),  with  a  drop  from  1.5  to  0.8.  The  count  in 
KAELOC  is  used  to  normalize  any  differences  in  the  development  languages  used.  As  seen  in 
the  figure,  the  trends  in  KLOC  and  KAELOC  are  quite  similar. 

Cost  of  quality  (COQ)  at  MIEL  has  declined  by  15  percent  since  2003  even  when  expressed 
as  fractions  of  100  percent  (Figure  29).  There  was  a  30  percent  decline  in  cost  of  quality 
when  expressed  as  a  decrease  from  the  baseline  of  50  percent  in  2003  to  just  under  35  percent 
in  2005.  Much  of  the  decrease  is  due  to  automation,  which  took  place  as  part  of  the 
institutionalization  of  improved  processes  based  on  the  Verification,  Validation,  and 
Organizational  Innovation  and  Deployment  process  areas.  The  slight  COQ  increase  in  2005  is 
due  the  inclusion  of  integration  and  interoperability  testing  as  MIEL  took  on  later  stage  life- 
cycle  products  and  services. 
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Figure  29:  Cost  of  Quality 

Motorola  Global  Software  Group  Centers  regularly  conduct  customer  satisfaction  surveys  at 
project  completion  for  every  project  the  organization  undertakes.  MIEL  ratings  have 
increased  from  an  average  of  8  (very  good),  at  the  end  of  the  SW-CMM  period,  to  9 
(excellent)  out  of  10,  since  its  move  to  CMMI.  The  survey  questions  query  explicitly  about 
categories  such  as  absence  of  defects,  communication,  domain  knowledge  of  engineers, 
functionality,  product  quality,  usability,  and  performance.  The  improvement  in  customer 
satisfaction  is  due  particularly  to  the  organization’s  increased  focus  on  quality  and 
innovation. 
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4.4.3  Motorola  GSG  Russia  Center 

written  with  Alexander  Babkin 


Background 

The  Organization  and  Its  Products 

Founded  in  1997,  Motorola’s  St.  Petersburg-based  software  development  center  is  part  of  the 
Motorola  Global  Software  Group  that  develops  software  for  Motorola  products.  The  GSG 
Russia  Center  is  one  of  the  major  software  development  facilities  in  the  Europe,  the  Middle 
East,  and  Africa  (EMEA)  region,  and  it  serves  Motorola  customers  worldwide.  Its  four  major 
domain  areas  include  wireless  platform  solutions,  automotive  and  telematics,  cellular 
networks  infrastructure,  and  access  solutions.  Currently,  the  Center  has  more  than  500 
employees. 

One  of  the  Center’s  competences  is  developing  software  for  integrated  cellular  and  trunking 
network  infrastructures.  This  work  includes  software  for  base  stations,  controllers,  latest- 
generation  mobile  switches,  and  other  components. 

The  Access  Solutions  department  specializes  in  remote  access  solutions,  and  it  is  a  leading 
software,  service,  and  hardware  provider  for  networking,  embedded  systems,  and  custom 
communications  applications. 

The  Center  also  works  on  real-time  embedded  systems.  These  systems  are  used  to  implement 
operating  systems  for  various  applications,  optimize  algorithms  using  special  functionality  of 
Motorola  processors,  and  analyze  real-time  system  behavior. 

An  important  field  for  the  Center  is  the  development  of  software  for  automotive  electronics 
and  telematics  systems.  The  Center  also  develops  software  functionality  for  Java-based 
handsets,  including  audio  and  video  reception,  processing,  and  transmission. 

As  seen  in  Figure  30,  almost  two  thirds  of  the  development  work  done  in  2004  and  2005  by 
the  GSG  Russia  Center  involved  prototyping  prior  to  the  full  implementation  of  new  products 
and  system  functionality.  Work  classified  as  “prototyping”  at  GSG  Russia  also  includes  new 
product  development  once  the  requirements  have  been  stabilized.  Several  rounds  of 
prototyping  typically  are  conducted  to  elicit  customer  requirements  and  get  early  feedback  on 
basic  system  functionality,  after  which  the  projects  proceed  with  further  development  of  the 
new  systems  and  functionality  for  the  delivered  products. 

About  half  of  the  Center’s  work  is  new  development  (see  Figure  31),  and  almost  40  percent  is 
devoted  to  maintenance  and  enhancement  of  existing  software.  The  “hybrid”  category 
includes  concurrent  repairs  and  adding  major  new  functionality  to  the  existing  software. 
“Porting/Localization”  efforts  involve  the  migration  of  existing  software  to  new  platforms  or 
user  interface  languages. 
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Figure  30:  Distribution  by  Project  Lifecycle  Models 


Figure  31:  Distribution  by  Project  Nature 

Motorola  GSG  Russia  measures  project  size  by  the  amount  of  what  they  call  delta  code.  As 
seen  in  Figure  32,  the  amount  of  such  code  that  was  added,  deleted,  or  modified  varied 
considerably  in  2004  and  2005;  the  projects  also  differed  considerably  in  terms  of  the  amount 
of  staff  time  they  require  (see  Figure  33). 
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Distribution  by  Project  Size  (Delta  KLOC) 
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Figure  32:  Distribution  by  Project  Size 


Figure  33:  Distribution  by  Project  Effort 

The  projects  can  differ  in  difficulty  regardless  of  product  size  and  effort  required.  For 
example,  some  projects  with  less  than  5KLOC  involve  simple  enhancement  of  existing 
functionality;  others  deal  with  difficult  optimization.  The  Russia  Center  uses  a  series  of 
“integral”  characteristics  to  help  project  managers  tailor  their  processes  as  appropriate.  The 
characteristics  are  obtained  after  weighting  the  projects  against  parameters  such  as  size, 
novelty  of  technologies,  complexity  of  interaction  within  a  team,  and  intended  use  of  the 
application.  Similar  procedures  are  used  for  calculation  of  cycle  time  reduction  metrics. 
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Process  Improvement  History 

Process  improvement  at  the  GSG  Russia  Center  has  been  guided  largely  by  Capability 
Maturity  Models,  initially  the  SW-CMM  and  then  CMMI.  Major  events  in  the  Center’s 
history  include  the  following. 

•  June  1997  -  start  of  deployment  of  software  production  processes 

•  July  1999  -  assessed  at  SW-CMM  maturity  level  3  using  the  CMM  Appraisal  Framework 
(CAF)  compliant  Motorola  Software  Assessment  method  and  by  doing  so,  became  the 
first  company  in  Russia  with  an  appraised  CMM  maturity  level 

•  September  2001  -  assessed  at  SW-CMM  maturity  level  5  using  the  CAF-compliant 
Motorola  Software  Assessment  method  and  by  doing  so,  became  the  first  company  in 
Russia  assessed  at  the  highest  maturity  level 

•  November  2002  -  start  of  CMMI  Transition  project 

•  October  2003 -April  2004  -  participation  in  piloting  of  SCAMPI  Class  C  and  B  Appraisal 
Methods 

•  November,  2004  -  appraised  at  CMMI  maturity  level  5  using  the  SCAMPI  Class  A 
method  and  by  doing  so  became  the  first  center  of  Motorola  world-wide  that  went 
through  a  full  succession  of  SCAMPI  Class  C,  B  and  A  appraisals 

•  2005-2006  -  beginning  CMMI  VI  .2  piloting 

Like  other  Motorola  GSG  Centers,  the  GSG  Russia  Center  also  has  used  Digital  Six  Sigma. 
Beginning  in  2003,  the  organization  used  aspects  of  that  methodology  to  implement 
processes  based  on  the  CMMI  Organizational  Innovation  and  Deployment  process  area.  GSG 
Russia  also  had  a  long  history  using  Goal-Question-Metric  (GQM)  methods  to  guide  its 
measurement  efforts.  With  those  exceptions,  GSG  Russia  has  implemented  no  other  major 
improvement  initiatives. 

CMMI-Based  Process  Improvement 

The  Motorola  GSG  Russia  Center  already  had  a  well-defined  and  proven  software  production 
process,  having  been  appraised  at  maturity  level  5  under  the  SW-CMM  for  over  a  year  prior 
to  initiating  its  transition  to  CMMI.  The  following  points  were  critical  in  GSG  Russia’s 
choice  to  upgrade  its  organizational  processes. 

•  increased  emphasis  in  CMMI  on  organization-level  business  goals 

•  incorporation  of  additional  industry  best  practices  in  the  Project  Management  process 
areas 

•  integration  of  additional  engineering  discipline  addressing  both  system  and  software 
production  processes 

•  flexibility  to  focus  on  selected  process  capability  profiles  using  the  CMMI  continuous 
representation 
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•  increased  clarity  provided  by  more  detailed  descriptions  and  guidelines  than  were  found 
in  the  CMMI  source  models 

•  increased  consistency  since  many  model  practices  were  significantly  reworked  to  remove 
duplications,  ambiguities,  and  cyclic  references 

•  upcoming  sunsetting  of  the  SW-CMM 

GSG  Russia  introduced  several  major  changes  to  its  software  production  process  during  its 

transition  from  the  SW-CMM  to  CMMI.  GSG  Russia  redefined  all  of  the  practices  described 

below  based  on  the  additional  explicit  guidance  it  found  in  CMMI. 

•  The  focus  on  business  objectives  in  CMMI  led  GSG  Russia  to  change  its  practices  for 
selecting  subprocesses  for  quantitative  and  statistical  control.  Moreover,  the  emphasis  in 
the  Measurement  and  Analysis  process  area  on  aligning  measurement  objectives  and 
activities  with  business  information  needs  and  objectives  helped  GSG  Russia  redevelop 
its  approach  to  managing  quantitatively  based  on  its  business  priorities. 

•  GSG  Russia  has  developed  and  deployed  process  performance  models  to  forecast  and 
control  cost  of  quality  (COQ)  and  cost  of  poor  quality  (COPQ)  indicators.  It  started 
development  on  more  formal  models  due  to  its  reading  of  CMMI,  particularly  the 
Organizational  Process  Performance  (OPP)  process  area.  This  Center  now  leverages  the 
use  of  historical  database  and  process  performance  baselines  to  predict  anticipated  values 
of  COQ  and  COPQ  based  on  classifications  by  type  and  complexity  of  projects. 

•  The  unification  in  CMMI  of  technology  and  process  change  management  approaches  to 
identify,  evaluate,  select,  and  deploy  significant  innovations  was  very  helpful  for  the 
GSG  Russia  Center.  The  Center  found  it  valuable  to  realign  what  previously  were  two 
separate  GSG  Russia  processes  into  a  single  more  focused  one  because  processes  and 
technologies  typically  are  implemented  together  in  a  unified  manner. 

•  GSG  Russia  shifted  its  focus  on  causal  analysis  processes  from  the  relatively  narrow 
defect  prevention  perspective  in  the  SW-CMM  to  the  broader  problem  prevention  focus 
in  CMMI,  and  now  addresses  issues  related  to  process  performance,  customer 
satisfaction,  and  process  compliance.  GSG  Russia  continues  its  proactive  focus  on  causal 
analysis  for  prevention  as  well  as  resolution  of  existing  problems,  which  it  attributes  to  a 
“high  maturity”  risk  management  perspective  in  its  reading  of  the  SW-CMM. 

•  Based  on  the  Decision  Analysis  and  Resolution  process  area  (DAR),  the  Russia  Center 
has  implemented  formal  decision  making  techniques.  Alternatives  for  selecting 
architectures  are  limited  for  its  products  that  are  embedded  into  larger  systems;  however, 
the  Center  has  used  DAR  techniques  to  evaluate  alternative  technical  solutions.  In 
addition,  this  Center  uses  life-cycle  modeling  to  update  organizational  processes  to 
accommodate  advanced  technologies  that  provide  alternative  solutions  that  otherwise 
would  be  difficult  to  apply  to  certain  categories  of  projects.  Using  outside  suppliers  is 
new  to  GSG  Russia,  so  it  also  formally  evaluates  alternatives  for  supplier  selection.  Other 
applications  of  DAR  include  decisions  about  alternative  process  and  technology  choices 
(e.g.,  scenarios  for  introducing  new  metrics  into  projects). 


CMU/SEI-2006-TR-004 


63 


•  GSG  Russia  has  developed  more  advanced  product  integration  procedures  as  a  result  of 
its  interpretation  of  CMMI.  In  particular,  the  Center  has  redesigned  its  tailorable  process 
integration  sequences,  and  alternative  sequences,  to  get  projects  thinking  about  product 
integration  procedures  from  the  very  beginning  so  that  elaborations  later  are  more 
manageable. 

•  This  Center  also  has  extended  its  use  of  operational  scenarios  to  include  requirements 
development  and  validation.  This  work  was  suggested  by  the  initial  CMMI  gap  analysis. 
Previously,  operational  scenarios  were  not  formalized,  and  they  were  considered 
separately  from  requirements.  Integrating  the  two  has  been  especially  helpful  for 
products  that  are  parts  of  larger  systems. 

•  Finally,  the  GSG  Russia  Center  has  revised  its  supplier  management  processes  to 
accommodate  CMMI  practices  related  to  the  selection  of  suppliers,  COTS  and  in-house 
vendor  reviews,  supplier  product  acceptance,  and  adoption. 

The  Motorola  GSG  Russia  Center  conducted  its  transition  to  CMMI  like  any  other  project  in 
Organizational  Innovation  and  Deployment  (OID)  fashion.  All  in  all,  more  than  30  percent  of 
the  organization’s  process  assets  were  revised  during  the  transition  project,  and  more  than 
another  10  percent  of  the  organization’s  process  assets  were  newly  created  based  on  CMMI. 
Approximately  20  staff  months  were  devoted  to  the  initial  gap  analysis,  process  changes,  and 
process  training.  It  took  14  additional  staff  months  for  organizing  and  conducting  subsequent 
SCAMPI  Class  C,  Class  B,  and  Class  A  appraisals.  It  took  nine  months  to  progress  from  the 
initial  gap  analysis  to  the  start  of  the  first  pilots  of  the  new  processes.  The  overall  duration  of 
transition  project,  from  the  initial  gap  analysis  through  the  SCAMPI  Class  A  appraisal,  was 
two  years. 

Performance  Results 

At  the  time  of  this  writing,  Motorola  GSG  Russia  Center  has  been  working  in  accordance 
with  revised  processes  based  on  CMMI  for  well  over  one  year.  During  that  time,  it  has  run  a 
series  of  improvement  projects  in  accordance  with  the  OID  process  area.  The  projects 
addressed  the  following  aspects  of  the  software  production  process: 

•  Automation  of  the  previously  existing  peer  review  process,  using  a  tool  for  planning, 
capturing,  and  reporting  review  results:  The  tool  works  remotely  with  a  focus  on 
identifying  high  priority  items  for  corrective  action. 

•  Automation  of  a  stakeholder  commitment  process,  which  facilitates  getting  explicit 
signoffs,  resource  allocations,  and  related  assurances  of  involvement 

•  Optimization  of  project  documentation  and  work  product  flow,  with  the  following  revised 
procedures  and  related  process  assets  to  reduce  redundancies: 

-  Collaborative  procedures  for  identifying  which  assets  (critical,  optional,  or 
informative)  must  be  shown  to  customers,  management,  product  teams,  or  other 
relevant  stakeholders  to  ensure  that  the  right  documents  go  to  the  right  people 
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-  A  series  of  project  templates  (e.g.,  for  testing,  design,  and  project  planning)  with  a 
focus  on  priorities  where  templates  are  most  necessary 

-  Redeveloped  audit  processes  to  catch  non-compliances  as  early  as  possible,  and, 
thereby,  reducing  the  total  number  of  audits  and  checks  without  compromising 
quality 

The  Russia  Center  has  significantly  improved  its  business  value  in  many  ways  based  on  its 
application  of  CMMI.  Key  performance  indicators  demonstrate  that  the  Center  has  both 
sustained  and  improved  the  performance  gains  it  made  based  on  the  SW-CMM.  The  Center 
has  significantly  reduced  its  cost  of  quality  and  markedly  shortened  its  product  cycle  time 
while  maintaining  compliance  with  the  organization’s  Six  Sigma-based  product  quality 
criteria.  The  following  results  were  obtained  from  more  than  30  projects. 

The  effort  counted  as  cost  of  quality  by  GSG  Russia  includes  verification  activities,  and 
particularly  peer  review  and  testing;  cost  of  quality  also  includes  quality  auditing  and 
problem  prevention.  Costs  of  poor  quality  include  rework  throughout  the  life  cycle.  Both  cost 
of  quality  and  cost  of  poor  quality  are  expressed  as  percentages  of  overall  effort  spent  for 
software  product  development.  As  seen  in  Figure  34,  cost  of  quality  is  approximately  half  of 
what  it  was  at  the  beginning  of  the  series  during  the  SW-CMM  period.  It  has  dropped  another 
third  from  its  value  when  the  Center  began  its  transition  to  CMMI.  Peer  review  effort  alone  is 
half  what  it  was  at  the  transition.  Cost  of  poor  quality  remained  under  5  percent  throughout 
the  entire  period. 


Cost  of  Quality/Cost  of  Poor  Quality 


— o— COQ%  —==COPQ% 


Figure  34:  Cost  of  Quality  and  Cost  of  Poor  Quality 

By  the  end  of  2005,  product  cycle  time  had  been  reduced  by  almost  40  percent  from  its 
baseline  during  the  Center’s  use  of  the  SW-CMM.  Well  over  half  of  that  improvement  took 
place  after  the  implementation  based  on  CMMI  (see  Figure  35).  Cycle  time  for  obtaining 
sign-off  commitments  was  cut  by  more  than  half  since  2002. 
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Figure  35:  Product  Cycle  Time  Reduction  Rate 

The  post-release  defect  rate  reported  by  GSG  Russia  customers  was  notably  low  from  the 
beginning  of  the  Center’s  adoption  of  the  SW-CMM,  and  it  was  even  lower  by  the  time  it 
moved  to  CMMI.  Since  the  upgrade  to  CMMI,  this  defect  rate  has  dropped  by  another  15 
percent  from  the  2002  baseline.  As  already  noted,  the  Center  has  sustained  this  low  post¬ 
release  defect  rate  while  significantly  improving  cost  of  quality  and  decreasing  cycle  time 


Post- Re  lease  Defects  Rate 


Figure  36:  Post-Release  Defect  Rate  (per  KLOC) 

The  return  on  investment  (ROI)  from  GSG  Russia’s  optimization  of  documentation  flow  was 
1.7:1  within  the  first  quarter  after  deployment.  Less  labor  effort  was  spent  on  projects  that 
produce  more  in  less  time.  Investments  counted  in  the  calculation  included  all  effort  spent  on 
developing  and  deploying  the  improvements,  which  included  new  training.  Measures  of 
return  included  reductions  in  cost  of  quality  and  cycle  time  based  on  labor  effort  only.  All 
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comparisons  were  normalized  using  baseline  parameters  for  the  type  of  product  produced. 
This  conservative  estimate  of  ROI  does  not  account  for  additional  business  value  made 
possible  by  the  Center’s  improved  efficiency.  For  example,  reductions  in  cost  of  poor  quality 
(COPQ)  were  not  counted  in  the  ROI  calculations  because  COPQ  is  well  under  control  in  this 
organization. 

The  Russia  Center  also  conducts  regular  surveys  of  its  process  practitioners’  satisfaction.  The 
results  improved  significantly  after  deploying  the  revised  CMMI-based  processes  (from  8.8 
to  9.2  out  of  a  possible  10  points). 

4.5  Raytheon  Network  Centric  Systems,  North 
Texas 

written  with  Jill  Brooks 


Background 

The  Organization  and  Its  Products 

Raytheon  is  an  industry  leader  in  defense  and  government  electronics,  space,  information 
technology,  technical  services,  and  business  aviation  and  special  mission  aircraft.  This  case 
description  focuses  on  one  of  the  organizations  that  form  a  part  of  the  larger  Raytheon 
Company. 

Raytheon  Network  Centric  Systems  (NCS),  which  is  headquartered  in  McKinney,  Texas, 
develops  and  produces  mission  solutions  for  networking  and  command  and  control.  The 
organization’s  programs  include  civilian  applications,  command  and  control  systems,  battle 
space  awareness,  integrated  communications  and  air  traffic  management  systems  and  netted 
sensor  systems.  NCS  serves  all  branches  of  the  U.S.  military,  the  National  Guard,  the 
Department  of  Homeland  Security,  the  Federal  Aviation  Administration,  other  U.S.  national 
security  agencies,  and  international  customers.  There  are  five  major  sites  and  numerous 
smaller  satellites  which  make  up  Raytheon  Network  Centric  Systems.  These  sites  have  a  total 
of  11,500  employees  and  contributed  to  Raytheon’s  $21.9  billion  in  sales  in  2005.  NCS  work 
breaks  down  into  four  major  areas:  hardware  engineering  accounts  for  39  percent  of  the  work 
done,  systems  engineering  accounts  for  27  percent,  24  percent  is  allocated  to  software 
engineering,  and  10  percent  to  management  and  support. 

Process  Improvement  History 

Raytheon’s  NCS  North  Texas  Software  Engineering  organization  began  its  process 
improvement  journey  in  1989,  when  it  was  part  of  Texas  Instruments.  In  the  12  years 
between  1989  and  2001,  the  organization  worked  on  developing,  deploying,  and  improving 
the  software  development  processes.  In  1992  as  part  of  Texas  Instruments,  the  organization 
was  presented  with  the  Malcolm  Baldridge  award  and  achieved  SW-CMM  maturity  level  2. 


CMU/SEI-2006-TR-004 


67 


In  1994,  the  organization,  which  was  still  a  part  of  Texas  Instruments,  achieved  SW-CMM 
maturity  level  3.  Beginning  in  1997,  Raytheon  merged  with  portions  of  Texas  Instruments, 
Hughes  Electronics,  and  E-Systems.  At  this  time,  the  new  organization  took  on  the  task  of 
bringing  together  processes  and  practices  from  the  three  separate  companies,  workforces,  and 
cultures.  To  help  the  organization  meet  this  challenge,  Raytheon  began  the  adoption  and 
development  of  Raytheon  Six  Sigma™.  In  June  2001,  Raytheon  applied  its  Six  Sigma 
practices  to  the  adoption  of  SW-CMM  and  achieved  maturity  level  4. 

CMMI-Based  Process  Improvement 

Raytheon  NCS  North  Texas,  in  conjunction  with  Raytheon  SAS  (Space  and  Airborne 
Systems)  North  Texas,  was  the  first  Raytheon  organization  and  the  5th  company  in  the  world 
to  achieve  CMMI  maturity  level  5.  In  September  2003,  it  achieved  CMMI  maturity  level  5  in 
its  software  organization  and  maturity  level  3  in  its  systems  organization.  Raytheon  NCS 
attributes  a  large  part  of  its  continued  success  to  the  application  of  its  Six  Sigma  program  to 
CMMI  implementation.  Another  important  success  factor  at  NCS  relates  to  its  focus  on 
change  management.  After  the  identification  of  a  performance  gap,  a  cross-functional,  multi¬ 
site  team  develops  a  “behavior  management  package”  that  includes  process  documentation 
that  is  deployed  to  all  engineering  personnel.  Multiple  techniques  are  used  to  facilitate  the 
desired  behaviors  which  are  followed  up  with  management  and  independent  reviews. 

Raytheon  NCS  introduced  a  number  of  new  practices  as  it  moved  from  the  SW-CMM  to 
CMMI  maturity  level  5  in  the  software  organization: 

•  The  organization  instituted  practices  based  on  the  Requirements  Development  process 
area,  made  modifications  to  practices  based  on  the  Requirements  Management  process 
area,  and  began  to  use  these  practices  to  balance  requirements  on  different  projects.  The 
enhanced  requirements  practices  are  used  to  assure  that  everyone  has  a  shared 
understanding  of  what  is  the  most  important  work,  to  identify  what  can  drop  off,  if 
necessary.  These  priorities  are  also  coordinated  with  customer  priorities  and  constraints. 

•  As  part  of  quantitative  project  management,  Raytheon  increased  its  use  of  statistical 
process  control  (SPC)  techniques  to  manage  its  peer  review  processes.  Tooling  was 
developed  to  facilitate  near  real-time  analysis  by  peer  review  participants.  Corrective 
actions  were  undertaken  in  instances  where  defects  were  above  or  below  established 
control  limits. 

•  NCS  applied  Causal  Analysis  and  Resolution  practices  at  the  organizational  level  to 
resolve  issues  and  contribute  to  the  Organizational  Innovation  and  Deployment  process. 

•  NCS  practices  based  on  the  CMMI  Decision  Analysis  and  Resolution  process  area  were 
applied  to  help  the  projects  in  the  organization  choose  among  alternative  process  and 
technical  solutions.  The  use  of  a  tool  called  Expert  Choice  [Expert  Choice  06],  which 
allows  users  to  identify  and  weight  criteria  and  assess  various  proposed  solutions  against 


™  Six  Sigma  is  a  registered  trademark  of  Motorola,  Inc. 
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these  criteria,  facilitated  the  implementation  of  DAR.  The  combination  of  this  tool  with 
the  Decision  Analysis  and  Resolution  process  area  was  cited  as  a  strength  in  the  NCS 
CMMI  appraisal. 

•  The  processes  employed  for  cost  estimation,  verification,  and  testing  were  refined. 

•  During  this  time  frame,  NCS  performed  many  Six  Sigma  projects  for  elements  of 
behavior  associated  with  most  of  the  organization’s  process  development. 

Combining  CMMI  and  Raytheon  Six  Sigma,  NCS  uses  the  following  steps  to  apply  CMMI 
maturity  level  4  and  5  process  areas  in  the  software  part  of  the  organization: 

•  a  process  capability  baseline  (Organizational  Process  Performance)  is  established 

•  goals  related  to  performance  excellence  are  established  and  regularly  monitored  for  the 
programs 

•  measurement  is  used  to  stabilize  processes 

•  if  this  baseline  does  not  satisfy  the  goals,  the  root  causes  (Causal  Analysis  and 
Resolution)  are  determined 

•  opportunities  for  improvement  are  then  identified,  selected,  prioritized  and  piloted 
(Organizational  Innovation  and  Deployment) 

•  if  these  changes  generate  improvements,  the  new  process  is  measured,  the  goals  are 
adjusted,  and  a  new  capability  baseline  is  established 

Following  these  successes,  NCS  faced  another  tough  cultural  and  organizational  issue. 
Historically,  the  organization  had  treated  its  systems  engineering  group  and  its  software 
group  as  different  organizational  entities.  Software  engineering  tends  toward  more  traditional 
development,  while  systems  engineering  focuses  more  on  creating  early  prototypes  and  on 
early  development  work.  The  fact  that  these  two  organizations  had  been  managed  separately, 
had  their  own  management  sponsorship  and  had  developed  their  own  set  of  processes  was  the 
reason  for  the  separate  appraisals. 

At  present,  these  groups,  including  hardware,  are  being  treated  as  one  organization  with  a 
common  process  architecture.  The  change  toward  integration  facilitates  the  shift  of  work 
packages  around  the  five  NCS  sites,  which  is  another  important  goal.  The  creation  of 
common  definitions  of  performance  measures  across  disciplines  has  been  a  major  challenge. 
In  addition  to  refining  and  extending  the  original  measures,  NCS  has  also  developed  a  tool  to 
help  gather  and  manage  the  data  collected  according  to  the  new  definitions. 

As  soon  as  NCS  has  completed  this  effort,  it  intends  to  have  a  joint  appraisal  to  evaluate  the 
entire  organization  (systems,  software  and  hardware)  against  all  five  levels  of  the  CMMI. 
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Performance  Results 


Raytheon  NCS  uses  its  measurement  results  to  characterize  performance  in  terms  of 
organizational  goals  and  to  identify  opportunities  for  improvement.  The  organization  is 
focused  on  the  following  four  requirements: 

•  meeting  budget  and  schedule  commitments 

•  providing  quality  products  that  satisfy  the  customer 

•  ensuring  that  its  pricing  remains  competitive 

•  providing  a  capable,  stable,  and  diverse  workforce 

The  organization’s  Cost  Performance  Index  (CPI)  and  Schedule  Performance  Index  (SPI) 
address  its  business  objectives  of  meeting  budget  and  schedule  commitments.  These 
measures  are  equally  applicable  to  software,  systems,  and  hardware  engineering. 

The  organization  addresses  its  goal  of  delivering  quality  by  measuring  defect  density.  As  a 
part  of  the  effort  to  develop  and  use  appropriate  defect  measures  for  different  disciplines, 
NCS  divided  the  development  effort,  in  general,  and  the  stages  identified  for  peer  reviews,  in 
particular,  into  14  stages.  Previously,  all  five  sites  used  different  phase  definitions  of  varying 
granularity.  Figure  37  shows  the  names  of  the  14  stages  and  the  measures  used  in  eight 
different  areas.  Only  two  of  the  measures  listed  below  the  areas  are  identical. 
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Figure  37:  Defect  Stages ,  Categories,  and  Measures 

Raytheon  NCS  also  measures  the  frequency  of  change  requests  (open  versus  closed)  and 
requirements  volatility.  The  latter  is  important  because  volatility  can  lead  to  higher  costs  than 
planned  and  to  related  risks  to  cost  and  schedule. 

Measurement  results  that  are  shared  in  this  report  date  back  to  the  2003  CMMI  maturity  level 
5  appraisal.  Due  to  the  organizational  changes  since  that  time,  NCS  has  updated  its 
measurements  so  that  they  are  consistent  across  the  organization  and  applicable  to  all  of  its 
disciplines.  Performance  results  based  on  the  new  measures  are  not  yet  available  and  analyses 
based  on  the  older  data  have  not  been  updated. 

Cost  and  Schedule  Performance  Indices  (CPI  and  SPI) 

The  Cost  Performance  Index  answers  the  question,  “How  much  work  has  been  completed  by 
this  point  in  time  compared  to  what  we  have  actually  spent  to  reach  this  point?”  Schedule 
Performance  answers  the  question,  “How  much  work  has  been  completed  by  this  point  in 
time  compared  to  what  was  scheduled  or  planned  to  be  completed?” 

A  major  contributor  to  improvements  and  predictability  in  cost  and  schedule  performance  is 
the  extent  to  which  projects  follow  standard  processes.  CPI  and  SPI  are  shown  using  very 
similar  charts.  As  seen  in  Figure  38,  NCS  was  able  to  improve  CPI  by  5  percent  between  the 
SW-CMM  maturity  level  4  appraisal  in  June  2001  and  the  CMMI  maturity  level  5  appraisal 
in  2003. 
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Figure  38:  Cost  Performance  Index 

Over  the  same  period  of  time,  variation  in  CPI  was  reduced  by  34  percent,  and  variation  on 
the  low  end  was  virtually  eliminated  (Figure  39). 
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Figure  39:  Cost  Variation 

Improvements  in  schedule  lead  to  an  8  percent  improvement  in  the  organization’s  schedule 
performance  index  and  a  50  percent  reduction  in  variation  between  June  2001,  when  the  last 
SW-CMM  appraisal  was  conducted  and  September  2003,  when  the  organization  was  awarded 
CMMI  maturity  level  5. 

Defect  Density  and  Defect  Containment 

Defect  density  is  used  in  conjunction  with  other  measures  to  monitor  a  program’s  progress 
toward  continuous  improvement  at  Raytheon  NCS.  In  2003  and  before,  defect  containment 
was  primarily  calculated  post  release.  The  quality  of  a  product  was  evaluated  by  the  number 
of  defects  per  delivered  unit  from  the  customer’s  point  of  view.  Defect  density,  compared  to  a 
desired  level  and  defect  containment,  evaluated  by  the  stage  of  injection  and  stage  of 
detection,  are  now  both  measured  using  the  newly  defined  project  phases.  Raytheon  NCS 
also  continues  to  measure  defect  variation  between  programs  and  total  defect  density  and 
defect  containment  numbers  in  qualification  testing  when  the  customer  is  present. 

It  is  more  difficult  to  attain  consistency  in  measures  of  defects  across  disciplines.  The 
organization  is  working  on  definitions  of  unit  measures  to  employ  in  different  disciplines  and 
contexts.  NCS  continues  to  use  source  lines  of  code  (SLOC)  as  the  primary  unit  for  software 
defects.  The  organization  found  it  necessary  to  define  different  defect  measures  for  software, 
systems  and  hardware,  which  include  the  following: 

•  number  of  interfaces  for  architecture  and  systems  integration 

•  numbers  of  specific  types  of  requirements  for  systems 

•  number  of  terminations  on  boards  for  hardware 
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•  lines  of  code  for  FPGA  (field  programmable  gate  arrays) 

•  number  of  test  requirements  for  test  engineering 

Figure  37  shows  all  of  the  measures  used  for  software,  systems  and  hardware. 

NCS  has  employed  Statistical  Process  Control  as  a  method  of  measuring  time  spent  on  peer 
reviews.  A  defect  density  measure  is  captured  and  used  to  determine  if  the  organization  needs 
to  conduct  additional  peer  reviews.  The  cost  of  fixing  each  defect  is  not  used  for  this  purpose 
because  NCS  prefers  to  focus  its  attention  on  correction  and  prevention  rather  than  on 
expenditures.  As  seen  in  Figure  37,  NCS’s  defect  density  in  software  was  reduced  by  44 
percentage  points,  and  the  variation  was  reduced  by  3 1  percent. 


Performance 


Improved  Defect  Density  by  44  percentage  points,  and  reduced  variation  by  31% 


Figure  40:  Defect  Density 

Raytheon  NCS  achieved  an  overall  ROI  of  3:1  through  significant  cost  avoidance  which 
resulted  from  its  organizational  improvements.  The  organization’s  overall  quality  program 
and  its  improvements  in  business  performance  were  recognized  as  organizational  strengths  in 
its  CMMI  maturity  level  5  appraisal. 
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4.6  TrialStat  Corporation5 

written  with  Khaled  El  Emam 


Background 

The  Organization  and  Its  Products 

TrialStat  Corporation  is  a  small  software  company  in  Canada  that  develops  software  for  use 
in  the  clinical  trials  phase  of  the  drug  development  process.  Its  customers  are  the 
pharmaceutical  industry  and  contract  research  organizations  that  run  clinical  trials  for  them. 
Its  products  are  also  used  in  observational  studies  such  as  patient  registries,  cohort  studies, 
and  disease  surveillance.  The  company  was  founded  at  the  end  of  2001 .  It  currently  has  over 
30  employees. 

TrialStat ’s  main  product  is  called  ClinicalAnalytics  (CA)  [TrialStat  06].  It  is  released 
iteratively  with  additional  functionality  added  for  various  customers.  The  software  runs  on 
multiple  platforms,  including  various  mobile  devices,  and  operates  in  connected  mode  (in  a 
Web  browser)  and  disconnected  mode  when  there  is  no  Internet  connectivity.  The 
development  team  consists  of  nine  developers,  one  requirements  analyst,  and  five  quality 
assurance  staff.  The  remainder  of  the  company  consists  of  the  data  center  operations  team, 
sales,  marketing,  professional  services,  support,  documentation,  and  executive  management. 

The  company  operates  in  a  regulated  environment.  The  most  relevant  FDA  regulations  are 
documented  in  the  Code  of  Federal  Regulations  (21  CFR  Part  11)  and  the  various  guidelines 
related  to  that  published  by  the  FDA  and  the  pharmaceutical  industry.  These  regulations 
apply  to  the  software  product  itself  and  to  the  processes  used  to  develop  and  maintain  that 
software.  In  addition,  because  the  software  application  is  used  to  collect  and  store  sensitive 
personal  health  information  on  thousands  of  individuals  from  around  the  world,  practices  to 
ensure  the  security  of  the  application  are  critical  and  are  usually  included  in  the  scope  of  a 
regulatory  audit. 

Process  Improvement  History 

TrialStat  has  been  involved  in  a  software  process  improvement  effort  since  2002. The  effort 
began  soon  after  startup  and  was  guided  initially  by  the  CMM  for  Software  and  then  by 
CMMI.  A  formal  CMMI-based  appraisal  has  not  been  performed,  but  the  CMM  models  have 
been  used  as  the  basis  for  a  continuous  internal  process  improvement  effort. 

Because  of  competitive  pressures,  the  release  cycle  for  the  CA  application  had  to  be  short. 
Therefore,  a  decision  was  made  early  to  adopt  an  agile  methodology  that  promised  rapid 
releases.  Proponents  of  agile  methods  recommend  a  three-week  (or  shorter)  release  cycle  and 

5  This  section  is  an  abridgement  and  extension  of  work  that  will  appear  in  the  forthcoming  book 
titled  CMMI :  Guidelines  for  Process  Integration  and  Product  Improvement,  Second  edition  by 
Mary  Beth  Chrissis,  Mike  Konrad,  and  Sandy  Shrum. 
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suggest  that  this  is  doable  in  practice.  At  the  outset,  a  three-week  release  cycle  was  attempted; 
however,  this  created  many  problems. 


A  three-week  release  cycle  resulted  in  rapid  burnout  of  the  development  team  as  the  pace 
became  increasingly  exhausting.  The  company  was  at  risk  of  losing  key  developers,  who 
were  unwilling  to  put  in  the  overtime  and  weekends  to  maintain  the  three- week  release  cycle. 
It  was  also  evident  that  the  only  way  to  have  such  short  iterations  was  to  curtail  most 
requirements  analysis  activities  and  to  have  absolutely  minimal  quality  assurance  on  the 
product,  both  of  which  were  unacceptable. 

The  development  team  then  experimented  with  increasing  the  release  interval.  After  a  number 
of  attempts  it  was  decided  that  a  three-month  interval  was  sufficient.  This  was  short  enough 
to  address  the  rapidly  changing  business  needs,  but  long  enough  not  to  exhaust  the 
development  team.  It  allowed  enough  time  for  sufficient  requirements  analysis  work  up  front 
and  for  effective  quality  assurance. 

All  process  improvements  were  (and  still  are)  implemented  within  this  three-month  iterative 
framework  as  described  in  the  next  section.  One  important  advantage  is  the  provision  of  rapid 
feedback,  making  it  possible  to  pilot  new  practices  and  tools  and  evaluate  their  value  quickly. 


CMMI-Based  Process  Improvement 

CMMI-based  practices  were  introduced  iteratively  at  TrialStat  in  conjunction  with  the 
releases  of  the  ClinicalAnalytics  products.  The  most  important  process  areas  for  TrialStat 
were  Organizational  Process  Focus,  Organizational  Process  Definition,  Process  and  Product 
Quality  Assurance,  Organizational  Training,  and  Measurement  and  Analysis.  The  initial  order 
of  implementation  focused  mostly  on  maturity  level  2  practices  to  meet  the  new  company’s 
recognition  of  the  importance  of  good  project  management.  At  the  same  time,  because  of  the 
regulated  nature  of  the  company's  business,  some  of  the  process-oriented  practices  in 
maturity  level  3  were  also  important  from  the  beginning. 

Resources  were  dedicated  to  defining  and  improving  processes  early  on  in  the  company’s 
history.  For  an  engineering  team  of  this  size,  a  part-time  allocation  of  a  senior  engineer  plus 
as-needed  administrative  support  was  sufficient  to  make  considerable  progress. 

A  strong  focus  on  process  definition  was  necessary  from  the  start.  Standard  operating 
procedures  documenting  all  of  the  engineering  and  business  processes  had  to  be  developed  at 
the  same  time  as  the  project  management  practices  were  being  implemented.  The  process 
definition  strategy  involved  experimenting  with  new  processes  until  they  were  working.  Each 
process  then  was  standardized  and  documented  in  a  standard  operating  procedure.  The  small 
organizational  size  meant  that  there  was  no  difference  between  organizational  and  project 
procedures;  they  were  all  the  same. 
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Process  documentation  proved  helpful  for  this  small  organization,  making  it  easier  to 
integrate  new  staff  and  ensure  that  the  staff  contributed  sooner.  Without  that  documentation, 
corporate  growth  would  have  been  more  painful.  The  people  typically  attracted  to  a  small 
organization  are  not  necessarily  process  oriented.  Process  documentation  contributed  to 
establishing  clear  ground  rules  for  new  staff  and  enforcing  a  process-oriented  corporate 
culture. 

Regular  internal  audits  ensure  process  compliance.  Audits  are  performed  by  third  parties. 
While  the  audits  are  against  the  FDA  regulations,  on  the  process  side  there  is  considerable 
overlap  with  CMMI  model  practices.  Training  capabilities  were  not  developed  in-house,  but 
were  all  outsourced.  However,  training  plans  and  records  have  to  be  maintained  for  all  staff 
as  part  of  the  company’s  regulatory  requirements. 

The  iterative  development  process  allowed  for  the  continuous  introduction  of  new  project 
level  practices  based  on  the  project  management,  engineering,  and  support  process  areas.  It 
also  enabled  rapid  feedback  on  the  effectiveness  of  these  practices.  Each  of  the  iterations 
represented  an  opportunity  to  introduce  new  processes,  a  new  technology,  or  expertise  in  the 
form  of  an  individual  with  specialized  skills.  After  three  months  it  was  possible  to  determine 
whether  the  intervention  succeeded  or  had  the  desired  impact.  If  it  did,  then  it  was  kept  for 
subsequent  iterations.  Those  that  cause  problems  were  either  adjusted,  taking  into  account 
what  was  learned,  or  eliminated  in  subsequent  releases. 

This  mode  of  introducing  changes  does  impose  some  constraints.  The  interventions  cannot  be 
large  because  the  development  team  has  to  be  able  to  learn  them,  master  them,  and  apply 
them  well  enough  in  the  iteration  to  provide  feedback  to  management  at  the  end  of  the 
iteration.  Therefore  new  practices  had  to  be  introduced  gradually.  For  example,  when  peer 
reviews  were  introduced,  they  focused  only  on  the  requirements.  Then  code  peer  reviews 
were  first  done  only  on  the  database  operations  that  were  likely  to  have  significant 
performance  impacts  on  system.  Then  they  were  extended  to  include  error  handling  code  as 
the  iterative  cycle  continued. 

Performance  Results 

The  Measures 

The  collection  and  use  of  measurement  for  decision  making  started  from  the  very  beginning 
of  the  project  and  was  subsequently  expanded  in  a  series  of  iterations.  First,  data  on  post¬ 
release  defects  were  collected.  These  data  were  necessary  to  manage  and  prioritize  defect 
correction  activities  and  resources.  Once  that  system  was  in  place,  measures  related  to  the 
ability  to  meet  schedule  targets  were  collected. 

Scope  management  was  the  next  issue.  Because  the  delivery  date  for  each  of  the  iterations 
was  fixed,  flexibility  was  necessary  to  control  the  scope  of  work  that  could  be  completed 
within  that  time  period.  Features  scheduled  for  the  iteration  were  sized  for  a  three-month 
cycle.  In  some  cases,  features  were  split  and  implemented  over  multiple  releases.  The 
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challenge  was  coming  up  with  an  appropriate  approach  to  measuring  the  size  of  the 
requirements  early.  Currently,  using  the  number  of  use  cases  to  measure  the  size  of 
requirements  has  worked  well  in  the  TrialStat  environment. 

The  measurement  of  size  and  complexity  became  the  subsequent  focus  of  measurement.  As 
the  system  grew,  it  became  critical  to  manage  its  size  and  complexity.  One  approach  used  was 
to  re-factor  specific  parts  of  the  system  to  reduce  complexity. 

The  resources  available  for  each  release  were  fixed.  The  same  number  of  people  was 
available  each  time,  and  each  release  cycle  had  a  fixed  duration.  Therefore,  the  estimation 
challenge  was  to  define  a  scope  that  was  small  enough  to  be  completed  within  the  available 
fixed  resources  and  time.  The  following  were  the  primary  determinants  of  whether  the  scope 
was  manageable: 

•  The  size  of  the  features  that  were  being  planned  for  a  release. 

•  The  complexity  of  the  features  that  were  being  implemented  for  a  release.  This  was 
determined  subjectively  by  the  number  of  new  screens,  the  number  of  new  database 
tables,  and  the  coupling  between  each  new  feature  and  existing  features. 

•  The  newness  of  the  technology  that  was  being  used.  New  technology  may  mean,  for 
example,  a  new  third  party  component  or  external  library,  a  new  data  exchange  standard, 
or  sub-discipline  such  as  genetics  that  needs  to  be  supported  by  the  ClinicalAnalytics 
product. 

•  The  availability  of  developers  who  are  most  suitable  for  a  feature.  For  example,  if  a  key 
database  developer  was  going  to  take  vacation  during  a  release  then  that  was  going  to 
have  an  influence  on  the  scope  that  is  scheduled  for  that  release. 

The  Results 

Because  of  the  continuous  process  improvement  throughout  most  of  the  history  of  the 
company,  TrialStat’s  goal  was  to  achieve  stable  or  incrementally  improving  performance 
results  over  time.  The  size  and  complexity  of  the  ClinicalAnalytics  product  has  increased 
with  each  of  its  iterations,  so  improvements  in  both  process  and  product  are  needed  just  to 
maintain  current  performance  status.  The  differences  in  size  and  complexity  over  time  also 
make  it  impossible  to  directly  compare  the  performance  results  before  and  after  each  process 
intervention;  however,  the  performance  results  can  be  compared  to  norms  and  benchmarks  to 
interpret  the  relative  value  added  by  the  CMMI-based  process  improvements. 

The  ability  to  meet  schedule  commitments  is  crucial  in  TrialStat’s  business.  Figure  41  shows 
the  company’s  ability  to  meet  the  deadlines  for  five  recent  major  releases  of  the  CA  software 
product.  The  y-axis  shows  the  delay  in  delivering  each  release  in  days.  Three  of  the  five  were 
delivered  on  time.  The  longest  delay  was  one  week  in  release  6,  which  represents  an 
approximately  8  percent  deviation  from  plan.  The  other  was  three  days  late. 
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Figure  41:  Ability  to  Meet  Schedule  Targets 

The  results  in  Figure  42  show  the  post-release  defect  density  for  a  series  of  recent  major  and 
minor  releases  of  CA  product  distributed  across  the  x-axis.  Defects  are  expressed  per  function 
point  on  the  y-axis.  These  defects  are  known  to  have  existed  in  the  product  post-release,  and 
were  discovered  through  customer  usage  or  internal  testing  after  the  product  was  deployed. 
Size  was  converted  to  function  points  using  Jones’  backfiring  table  [Jones  00]. 

The  line  at  the  top  of  the  graph  is  a  benchmark  for  projects  in  the  manufacturing  domain. 
From  the  most  recently  published  data  set  of  the  International  Software  Benchmarking 
Standards  Group6  (ISBSG),  manufacturing  had  the  highest  median  quality  benchmark  across 
all  of  the  domains  reported  in  The  ROI  from  Software  Quality  [El  Emam  05].  As  can  be  seen 
in  Figure  42,  while  there  is  variation  in  quality  across  the  releases,  the  defect  density  per 
release  tends  to  be  considerably  lower  than  the  benchmark.  The  peaks  in  the  graph  coincide 
with  the  largest  releases,  and  the  troughs  with  the  smallest  releases. 

Organizations  that  contribute  data  to  the  ISBSG  benchmarks  tend  to  submit  results  from  their 
better  projects,  so  this  benchmark  overstates  average  performance.  It  also  understates  the 
difference  with  the  TrialStat  defect  density  results. 


6  ICBSG  is  an  international  organization  based  in  Australia  that  collects  data  from  software  projects 
around  the  world  and  produces  various  software  project  benchmarks  on  an  ongoing  basis 
(http://www.isbsg.org/). 
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Figure  42:  Post-Release  Defect  Density 

The  results  in  Figure  43  translate  the  quality  advantage  compared  to  the  benchmark  into 
dollar  figures  for  end  users  of  the  CA  software.  The  estimated  cost  savings  shown  in  the 
figure  accrue  due  to  a  lower  incidence  of  production  and  shipment  delays,  less  system  down 
time,  increased  customer  confidence,  customer  satisfaction  with  respect  to  timing,  and  fewer 
lost  clients  by  using  a  higher  quality  product;  the  results  assume  a  100-person  company  at  an 
average  cost  due  to  product  defects  of  $1,466  per  employee  when  using  benchmark  quality 
products  other  than  ClinicalAnalytics  [Tassey  03].  A  simple  model  to  estimate  these  savings 
is  presented  in  more  detail  in  The  ROI from  Software  Quality  [El  Emam  05]. 

The  figure  shows  the  estimated  cost  savings  of  each  using  each  of  the  ten  ClinicalAnalytics 
releases  compared  to  using  another  product  that  had  the  same  quality  level  as  the  ISBSG 
benchmark.  For  example,  a  100-employee  contract  research  organization  that  uses  release  E 
likely  would  experience  a  saving  of  more  than  $100,000.  Such  dollar  savings  provide  a 
compelling  reason  for  TrialStat’s  customers  to  continue  using  the  company’s  software. 
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Figure  43:  Cost  Savings  to  the  End  User 


4.7  Tufts  Health  Plan 

written  with  Alan  Bruno  and  Rob  Houle 


Background 

The  Organization 

The  Tufts  Health  Plan  was  founded  in  1979  as  a  not-for-profit  health  maintenance 
organization  (HMO).  It  has  since  expanded  to  a  solution-based  health  plan  that  offers  a 
variety  of  health  care  coverage  options,  including  a  Preferred  Provider  Organization  (PPO), 
Point  of  Service  (POS),  Exclusive  provider  plan  (EPO),  five  consumer-driven  health  plans 
and  Medicare  Advantage  plans.  There  are  1800  employees  in  the  health  plan,  excluding 
doctors  and  allied  health  professionals  who  provide  care,  and  the  provider  network  serves 
about  620,000  members.  In  a  national  ranking  of  more  than  500  health  plans,  Tufts  Health 
Plan  was  named  third  in  the  nation  for  both  HMO  and  POS  products  and  second  in  New 
England  for  clinical  performance  and  member  satisfaction.  They  are  among  10  percent  of  all 
health  plans  which  have  earned  the  Excellent  accreditation  status  from  the  National 
Committee  for  Quality  Assurance  (NCQA)  NCQA  is  a  nationally  recognized,  independent, 
not-for-profit  organization  dedicated  to  improving  the  quality  of  health  care.  Tufts  Health 
Plan  Medicare  Preferred,  a  Medicare  Advantage  program  for  Medicare  beneficiaries,  was 
ranked  as  one  of  the  highest  rated  health  plans  in  the  nation.  Tufts  Health  Plan  is  also  the  first 
health  plan  in  Boston  to  earn  a  J.D.  Power  and  Associates  award  for  outstanding  member 
satisfaction  or  service  excellence  for  10  consecutive  years. 
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There  are  300  full  time  equivalent  employees  and  contractors  in  the  Information  Technology 
(IT)  department.  These  include  personnel  in  development,  operations,  help  desk,  telecom, 
support  and  business  process  re-engineering.  This  department  supports  the  multitude  of 
business  functions  required  to  manage  the  delivery  of  health  care  to  the  organization’s 
members  (e.g.,  claims  processing,  referrals,  information  access,  medical  care  management, 
plan  benefits  inquiries,  and  transaction  processing). 


Process  Improvement  History 

In  addition  to  many  quality  achievements  in  its  primary  health  related  services,  Tufts  Health 
Plan  IT  already  had  a  considerable  quality  management  framework  in  place  before  it 
embarked  on  SW-CMM  and  CMMI.  Its  Systems  Engineering  Process  Architecture  (SEPA) 
was  established  in  2000  to  develop  and  integrate  a  variety  of  systems  into  the  technology 
infrastructure.  This  process  architecture  covers  all  phases  of  system  development  and  is 
organized  around  well-defined  project  delivery  life  cycles  including  waterfall  and  rapid 
development.  The  term  system  explicitly  includes  considerations  of  infrastructure  and 
hardware.  Systems  are  either  fully  developed  in-house  or  in  partnership  with  subcontractors. 
They  include  Tufts  Health  Plan  proprietary  software  development,  the  integration  of 
commercial-off-the-shelf  (COTS)  components,  and  hybrids  of  the  two. 

Several  other  models  and  frameworks  were  employed  in  developing  the  SEPA.  The 
management  of  the  operational  infrastructure  has  been  based  on  the  standards  of  the 
Information  Technology  Infrastructure  Library  (ITIL).  Business  needs  assessments,  project 
planning,  and  project  management  practices  are  well  defined  within  the  organization 
following  the  Program  Management  Book  of  Knowledge  (PMBOK)  methodology. 

Having  accomplished  many  fundamental  improvements  already,  the  adoption  of  the  SW- 
CMM  was  an  evolutionary  step.  Tufts  Health  Plan  had  in  place  quality  programs  designed  to 
maintain  the  stability  of  its  processes,  infrastructure,  and  operations.  The  organization  had 
mature  quality  processes  in  the  areas  of  configuration  management,  project  management, 
software  quality  assurance,  and  change  management.  Tufts  Health  Plan  had  employed 
Architecture  Review  Teams  and  Change  Control  Boards  for  oversight  of  system  changes  for 
several  years  prior  to  the  exploration  of  SW-CMM.  IT  at  Tufts  Health  Plan  already 
understood  the  benefits  of  process  management,  measurement  and  analysis,  and  a  greater 
integration  between  the  development  project  and  support  project  teams,  particularly  during 
the  initiation  and  planning  phases  of  the  life  cycles. 

It  was  not  until  they  completed  a  formal  process  needs  assessment,  however,  that  the  process 
team  came  to  see  how  much  their  improvement  needs  coincided  with  the  content  of  the  SW- 
CMM.  The  SW-CMM  effort  began  in  2001  when  the  IT  organization  formalized  its  process 
management  area  and  reviewed  the  results  of  a  needs  assessment  to  quantify  the  lessons 
learned  from  its  most  recent  development  projects,  and  to  examine  root  cause  data  from  its 
formal  configuration  management  and  project  management  groups.  The  output  was  a  “punch 
list”  of  areas  needing  improvement,  prioritized  by  the  number  of  occurrences  and  by  their 
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criticality  to  the  development  process.  The  requirements  of  the  Organizational  Process 
Definition  and  Organizational  Process  Focus  key  process  areas  brought  more  structure  to  the 
maturing  Process  Management  Team  (PMT).  The  SW-CMM  offered  solid  guidance  on  the 
characteristics  of  mature,  stable  processes  and  assisted  in  defining  goals  and  objectives  for 
the  processes  that  the  organization  was  trying  to  improve. 

Tufts  Health  Plan  adopted  the  SW-CMM  in  three  years  while  still  running  day-to-day  IT 
operations.  The  formal  SW-CMM  assessment  took  place  in  December  2004  and  Tufts  Health 
Plan  achieved  a  maturity  level  3  rating. 

CMMI-Based  Process  Improvement 

Although  Tufts  Health  Plan  IT  was  aware  of  the  CMMI  before  2004,  it  decided  to  fulfill  the 
requirements  of  a  SW-CMM  assessment  before  initiating  work  on  CMMI.  Tufts  Health  Plan 
IT  began  work  on  the  CMMI  in  January  2005.  Their  decision  to  move  to  CMMI  was 
motivated  by  two  business  reasons:  1.  the  organization  could  no  longer  be  appraised  against 
SW-CMM  since  the  model  was  sunset  by  the  SEI,  and  2.  other  companies  were  beginning  to 
ask  about  CMMI  compliance  in  Requests  for  Proposals.  After  all  the  improvements  Tufts 
Health  Plan  had  undertaken,  the  organization  had  to  make  only  a  few  additions,  and  minor 
changes  to  implement  the  relevant  CMMI-based  process  improvements. 

The  organization’s  primary  CMMI  focus  was  on  the  following  process  areas: 

•  Requirements  Development  (RD):  Tufts  Health  Plan  already  had  a  fairly  complete 
requirements  development  process.  To  enhance  it,  the  people  in  the  process  group  added 
sign-offs  by  developers,  testers,  and  others  who  might  be  involved  in  peer  reviews  and 
with  customers.  This  added  surety  about  agreement  on  the  part  of  all  parties  before 
proceeding  into  development.  It  also  resulted  in  fewer  requests  for  enhancements,  which 
was  one  of  the  goals  by  which  the  organization  was  measuring  itself. 

•  Decision  Analysis  and  Resolution  (DAR):  Tufts  Health  Plan  developed  DAR  guidelines 
and  applied  these  wherever  it  used  DAR.  In  particular,  DAR  was  applied  to  the  risk 
management  and  the  software  selection  processes.  The  organization  had  a  detailed 
software  selection  process,  which  began  with  receiving  RFPs  from  vendors  and 
proceeded  through  evaluation  and  selection  of  a  system.  DAR  is  used  for  the  more 
difficult  tool  or  software  selections. 

•  Organizational  Process  Focus  (OPF),  Organizational  Process  Definition  (OPD)  and 
Organizational  Innovation  and  Deployment  (OID):  Tufts  Health  Plan  IT  first 
implemented  its  Process  Management  Team  (PMT)  as  it  began  to  implement  SW-CMM. 
The  PMT  has  improved  so  much  since  that  time,  the  appraisers  said  it  looked  like  a 
maturity  level  5  process.  The  Process  Management  Team  (PMT),  which  consists  of 
representatives  from  all  IT  departments,  meets  on  a  regular  basis  and  reviews  IT  policies 
and  procedures.  The  team  performs  impact  analyses  before  approving  any  changes  and 
communicates  all  approved  changes  via  e-mail  bulletins. 
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•  Risk  Management  (RSKM):  Tufts  Health  Plan  IT  made  adjustments  to  its  metrics  plans 
and  standard  templates  in  order  to  include  risk  management;  RSKM  metrics  are  now 
collected  and  analyzed  regularly. 

Tufts  Health  Plan  IT  achieved  maturity  level  3  of  CMMI  SE/SW  1.1  at  the  end  of  February 
2006.  Its  strengths  included  the  following: 

•  Metrics  reports  that  provide  management  with  a  clear  picture 

•  Tools,  especially  its  document  management  system  and  their  PAL 

•  Organizational  Process  Focus  and  Organizational  Process  Definition  (PMT) 

Performance  Results 

Tufts  Health  Plan’s  IT  organization  established  a  metrics  management  system  structured  for 
three  primary  focus  areas:  IT  value,  business  support,  and  IT  operations.  The  organization  has 
focused  its  effort  on  improving  two  specific  categories  within  the  IT  Operations 
measurements  area:  project  performance  and  process  performance.  Analyses  center  around 
tracking  historical  trends  and  comparing  actual  performance  against  estimates.  They  have 
further  focused  considerable  attention  on  quality  issues. 

Project  performance  measures  were  developed  by  deriving  measurement  goals  from  the 
stated  strategic  goal  of  improving  the  performance  of  system  development  projects.  The  aim 
was  to  provide  the  project  manager  with  timely  information  about  the  schedule,  cost,  and 
quality  status  of  the  product  under  development.  Cost  measurements  include  the  costs  for 
actual-to-planned  labor  as  well  as  software,  hardware,  and  other  expenditures.  The 
scheduling  category  includes  comparisons  of  actual-to-planned  work  completed,  and  actual- 
to-planned  effort  expended.  These  measurements  are  calculated  for  both  the  given  reporting 
period  and  the  cumulative  project  timeline,  and  they  must  be  generated  with  sufficient 
frequency  to  allow  the  project  manager  to  take  corrective  action. 

Process  performance  is  measured  at  the  organization  level  using  the  cumulative  results  of 
individual  project  metrics.  Measurements  about  the  organization's  commitment  to  managing 
and  improving  processes  are  added.  Specific  measurements  include  process  improvement 
volume,  actual-to-planned  effort  expended,  and  actual-to-planned  team  activities  completed. 

The  quality  category  includes  measurements  which  provide  broader  insight  into  the  operation 
of  key  processes.  These  are  chosen  because  their  execution  directly  influences  the  quality  of 
the  product  under  development.  Measurements  in  this  category  focus  on  requirements, 
testing,  peer  reviews  and  quality  reviews,  and  production  issues. 

Each  project  captures  a  minimum  standard  set  of  metrics  which  can  then  be  rolled  up  to  a 
higher  level.  Project  schedules  are  tracked  using  dashboards  that  are  reviewed  by  the 
Corporate  Project  Management  Office.  They  use  Service  Level  Agreements  to  measure  and 
track  customer  satisfaction.  Tufts  Health  Plan  is  in  the  process  of  developing  a  method  for 
calculating  their  return  on  investment  (ROI).  It  already  tracks  costs  of  fixes  after  test  and  next 
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steps  are  to  calculate  the  baseline  figures  of  costs  to  fix  defects  found,  and  to  calculate  ROI  in 
dollars. 

At  Tufts  Health  Plan  IT,  most  work  is  developed  for  internal  customers,  and  that  means  that 
the  most  important  measurements  relate  to  deploying  products  into  production.  Testing  is  one 
place  where  pre-production  defects  are  identified.  Defects  are  identified  during  unit, 
functional,  systems,  and  acceptance  test  cycles  which  are  run  in  the  test  processing 
environment.  As  shown  below,  -total  defects  -dropped  by  16  percent  -between  the  last  year 
when  the  organization’s  processes  were  based  on  the  SW-CMM  and  the  implementation  of 
CMMI  processes  (2004-2005).  Over  the  4  years  portrayed  below,  defects  dropped  by  44 
percent. 


2002 

2003 

2004 

2005 

2004-2005 

Defects  Identified  in  Testing 

2796 

2469 

1862 

1565 

-16% 

Figure  44:  Testing  at  Tufts  Health  Plan 

Peer  reviews  comprise  another  important  quality  check  at  Tufts  Health  Plan.  The  figures 
below  show  that  the  number  of  peer  reviews  increased  68  percent  between  2004  and  2005, 
and  the  number  of  issues  identified  increased  59  percent.  These  reviews  are  conducted  prior 
to  the  moving  the  software  or  system  into  the  production  environment.  Finding  issues  at  this 
point  means  that  they  it  costs  less  to  fix  problems  and  the  defects  are  removed  before  they 
can  be  seen  by  customers. 


2003 

2004 

2005 

2004-2005 

Total  Peer  Reviews 

- 

324 

545 

+68% 

Total  Issues  Found  in  Peer  Reviews 

- 

763 

1211 

+59% 

Figure  45:  Peer  Reviews  at  Tufts  Health  Plan 

Tufts  Health  Plan  IT  calculates  productivity  based  on  the  number  of  change  requests  they 
field  each  year.  When  more  time  is  spent  on  valued  added  change  requests,  the  organization 
is  more  productive.  All  work  comes  into  the  IT  department  via  change  requests-from  minor 
corrections  to  major  new  systems.  Change  requests  at  Tufts  Health  Plan  act  like  work  orders 
An  example  of  how  a  work  order  generates  both  small  and  large  changes  was  a  recent  work 
order  to  change  member  identification  numbers  from  Social  Security  numbers  to  random  ID 
numbers.  If  something  needs  to  be  fixed,  enhanced,  replaced  or  built  from  scratch,  it  comes 
into  the  department  through  a  change  request.  Corrective  change  requests  identify  problems 
which  need  to  be  fixed.  The  proportion  of  corrective  change  requests  remained  relatively 
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steady  as  a  percentage  of  the  total  work  to  be  done,  hovering  around  23  percent.  There  were 
15  percent  more  total  change  requests  between  2004  and  2005. 


2003 

2004 

2005 

2004-2005 

Total  Change  Requests  (All  Categories) 

1719 

1607 

1849 

+15% 

%  of  Corrective  Change  Requests 

23% 

24% 

22% 

-2% 

Figure  46:  Change  Requests  at  Tufts  Health  Plan 

Metrics  related  to  production  issues,  or  the  count  of  total  global  or  systemic  problems  that 
occur  in  the  production  processing  environment,  comprise  the  final  quality  measure 
addressed  in  this  report.  These  are  issues  that  come  to  light  after  software  or  a  system  has 
been  deployed  into  production.  These  issues  are  taken  very  seriously.  The  Production  Issue 
Management  Team,  a  cross  functional  group,  holds  daily  meetings  to  review  product  defects 
identified  from  the  previous  day.  In  some  cases,  correcting  the  problem  may  be  as  simple  as 
adding  a  new  field  to  a  checklist.  In  other  cases,  the  issue  is  more  complicated.  In  these  cases, 
members  of  the  team  conduct  a  root  cause  analysis  to  identify  the  source  and  recommend 
process  improvements  to  ensure  elimination  of  future  defects.  The  good  news  is  that  the 
number  of  Production  Issues  has  also  declined  in  the  last  two  years. 


2002 

200 

2004 

2005 

2004-2005 

Total  Production  Issues 

1550 

1306 

1182 

1086 

-8% 

Figure  47:  Production  Issues  at  Tufts  Health  Plan 

Tufts  Health  Plan  achieved  better  quality  products,  delivered  on  time,  and  within  budget,  to 
more  customers  who  were  satisfied  with  the  results.  In  addition,  100  percent  of  its  major 
projects  were  completed  on  time  in  2004  and  2005.  Perhaps  what  is  most  notable  is  that  the 
early  improvement  trends  continued  and  accelerated  during  the  deployment  of  the 
organization’s  CMMI-based  improvements.  The  results  are  summarized  below. 

•  16  percent  decline  in  defects  identified  in  testing 

•  68  percent  increase  in  the  number  of  peer  reviews 

•  59  percent  increase  in  the  total  issues  found  in  peer  reviews 

•  2  percent  decrease  in  corrective  change  requests 

•  15  percent  increase  in  the  total  number  of  change  requests  fielded 
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•  8  percent  decline  in  production  issues  (These  are  global  or  systemic  problems  in  the 
production  environment.) 

4.8  Warner  Robins  Air  Logistics  Center 

written  with  Millee  Sapp ,  Robert  Stoddard,  and  Thomas  Christian 

Background 

In  October  1997,  Warner  Robins  Air  Logistics  Center  (WR-ALC)  at  Robins  Air  Force  Base, 
Georgia,  consolidated  its  four  previously  separate  software  organizations  into  a  single 
division  called  the  Software  Engineering  Division.  The  consolidated  organizations  previously 
had  some  experience  using  the  SW-CMM  for  process  improvement,  but  these  activities  had 
been  performed  separately  in  each  of  the  original  four  organizations. 

A  leadership  decision  to  target  a  SW-CMM  maturity  level  3  rating  for  the  new  division 
required  enterprise-wide  process  standardization  to  ensure  a  consistent  management 
discipline  across  all  of  the  organization’s  business  domains.  As  a  result,  an  initiative  was 
established  to  consolidate  the  software  processes  of  the  original  four  organizations  into  a 
standard  set  of  processes  that  the  entire  organization  could  use.  The  initiative  defined  three 
major  domain  processes: 

•  Test  Program  Set  (TPS)  Development 

•  TPS  Maintenance  and  Modification 

•  Operational  Flight  Programs  (OFP) 

The  Software  Engineering  Division  immediately  established  an  integrated  Software 
Engineering  Process  Group  (SEPG)  and  a  Quality  Metrics  Group  (QMG)  to  begin  building 
upon  the  previous  software  process  improvement  experience  and  strengths  brought  together 
in  the  consolidation.  With  the  assistance  of  the  Software  Engineering  Institute  (SEI),  the 
SEPG  and  QMG  established  a  process  architecture  to  adopt  the  best  practices  identified  in  the 
existing  best  practices  from  the  four  original  organizations  and  map  those  practices  to  the 
SW-CMM.  This  architecture  became  the  design  document  for  the  corporate-  and  domain- 
level  process  documents  that  all  projects  in  the  organization  would  follow.  The  Software 
Engineering  Division  achieved  SW-CMM  maturity  level  3  in  an  SEI-led,  CMM-Based 
Appraisal  for  Internal  Process  Improvement  in  April  of  2000. 

During  this  same  timeframe,  WR-ALC  became  involved  in  the  CMMI  project.  WR-ALC 
participated  in  a  CMMI  pilot  appraisal  that  brought  together  projects  involved  in  systems 
engineering,  software  engineering,  and  software  acquisition. 

In  July  2001,  the  Software  Engineering  Division  decided  to  transition  from  the  SW-  CMM  to 
CMMI.  For  the  next  12  months,  the  SEPG  performed  a  gap  analysis  to  determine  which  areas 
of  the  organization’s  process  architecture  would  need  to  change.  The  priority  was  to  correct 


CMU/SEI-2006-TR-004 


87 


problems  or  make  enhancements  to  organizational  processes  based  on  the  CMMI  process 
areas  at  maturity  levels  2  and  3.  The  SEPG  also  began  to  identify  the  requirements  for 
achieving  maturity  levels  4  and  5. 

In  October  2002,  the  division  became  part  of  the  newly  established  Maintenance  Directorate 
(WR-ALC/MAS)  and  benefited  immensely  from  the  director’s  strong  support  for  software 
process  improvement.  In  December  2002,  the  division  rolled  out  a  revised  software 
engineering  process  to  the  organization.  Implementation  plans  once  again  spelled  out  when 
projects  within  the  organization  would  implement  these  new  processes. 

From  September  through  December  2003,  four  Standard  CMMI  Appraisal  Method  for 
Process  Improvement  (SCAMPI)  Class  B  appraisals  were  performed  to  assess  the 
organization’s  readiness  for  a  SCAMPI  Class  A  appraisal.  During  the  Class  B  appraisals, 
team  members  from  WRALC,  Ogden  ALC,  and  the  SEI  reviewed  more  than  980  artifacts 
across  seven  projects  and  identified  areas  of  risk  in  how  the  organization  was  implementing 
the  practices  identified  in  CMMI.  These  appraisals  led  to  action  plans  for  addressing  the  risk 
areas  and  verification  meetings  between  each  of  the  projects  and  the  SEPG  to  determine 
whether  the  organization  was  adequately  addressing  the  findings.  An  appraisal  in  October 
2004  confirmed  that  the  Software  Engineering  Division  had  achieved  CMMI  maturity  level 
5. 

With  the  advent  of  the  Air  Force  Materiel  Command  restructure,  the  Software  Engineering 
Division  has  reorganized  into  the  402d  Software  Maintenance  Group  (402  SMXG).  This 
reorganization  consolidated  the  existing  eight  branches  into  five  squadrons  and  one  group 
staff  organization.  The  402  SMXG  organization  continues  to  mature  its  processes  and  provide 
exceptional  products  and  support  to  its  customers. 

CMMI-Based  Process  Improvement 

Achieving  CMMI  maturity  level  5  would  not  have  been  possible  without  software  processes 
standardized  across  the  entire  715-person  Software  Engineering  Division.  In  addition  to  the 
large  number  of  software  personnel,  the  division  consisted  of  eight  branches  that  had 
different  focuses.  Nearly  500  of  the  personnel  have  electronics  engineering  and  computer 
science  degrees.  Each  of  these  branches — an  artifact  of  the  1997  consolidation  of  the 
software  organizations  supporting  the  directorates  for  avionics,  electronic  warfare,  the  F- 15, 
and  the  Joint  Surveillance  Target  Attack  Radar  System  (JSTARS) — had  different  customers 
and  its  own  individual  cultures.  Furthermore,  two  branches  worked  only  in  the  test  program 
set  development,  maintenance  and  modification  domains,  while  five  other  branches  worked 
only  in  the  OFP  domain.  The  eighth  branch  contained  the  SEPG,  Quality  Metrics  Group 
(QMG),  and  administrative  support  such  as  computer  network,  financial,  and  personnel 
specialists. 

The  keys  to  standardizing  processes  across  this  diverse  landscape  were  centralized  direction 
and  decentralized  execution.  The  SEPG  and  QMG  developed  the  organization’s  standard 
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software  processes,  following  a  standard  format.  These  documents  spelled  out  in  detail  a 
description  of  each  process  activity,  the  reason  for  it,  the  entry  criteria,  its  inputs  and  tasks, 
the  exit  criteria,  outputs,  measures  of  the  activity,  and  any  required  tailoring.  The  division’s 
adamant  commitment  to  excellence  resulted  in  several  detailed  documents.  These 
standardized  documents  provided  organizational  direction  and  policy;  thus,  they  laid  the 
foundation  for  the  organization’s  further  success. 

The  execution  of  division-wide  organizational  processes  was  decentralized  by  having 
functional  experts  from  each  branch  document  software  processes  at  the  domain  or  branch 
level.  Following  the  same  format — description,  reason,  entry  criteria,  inputs,  tasks,  exit 
criteria,  outputs,  measurement,  and  tailoring — these  documents  provide  the  standardization  of 
software  activities  within  a  domain  regardless  of  project  size,  complexity,  customer,  or 
priority. 

Substantial  improvements  took  place  in  the  organization’s  practices  that  were  heavily 
influenced  by  the  Causal  Analysis  and  Resolution  (CAR),  Organizational  Innovation  and 
Deployment  (OID),  Decision  Analysis  and  Resolution  (DAR),  and  the  CMMI  engineering 
process  areas.  Many  newly  detailed  processes  were  documented  for  the  division,  and  various 
plans,  guides,  and  other  process  assets  were  prepared.  The  following  items  are  among  those 
most  pertinent  for  the  performance  outcomes  presented  in  this  case  description: 

•  documents  describing  the  new  software  engineering  process 

•  the  peer  review  process 

•  practices  for  developing,  implementing,  and  maintaining  organizational  processes 

•  the  measurement  program  and  the  measurement  plan. 

Both  TPS  domains  tailored  their  processes  from  the  organizations  standard  processes.  Owing 
to  the  unique  focus  of  the  branches  in  the  Operational  Flight  Program  domain,  each  branch 
prepared  its  own  process  guide.  For  example,  in  the  JSTARS  branch,  the  public-private 
partnership  between  the  Northrop  Grumman  Corporation  and  WR-ALC  ensured  that 
processes  were  standardized  so  that  the  software  produced  at  the  two  different  locations 
(Robins  Air  Force  Base  and  Melbourne,  Florida)  could  be  integrated  readily.  Elsewhere,  the 
F-15  branch  standardized  its  processes  among  its  three  customers — the  Israeli  Air  Force, 
Royal  Saudi  Air  Force,  and  U.S.  Air  Force — to  benefit  all  three.  All  five  OFP  branches 
(avionics/airlift,  electronic  warfare,  F-15,  JSTARS,  and  Special  Operations  Forces/Combat 
Search  and  Rescue)  standardized  process  guides  in  accordance  with  the  division’s 
organizational  documents,  which  mandated  adherence  to  these  processes  by  all  of  the  OFP 
branches,  regardless  of  weapon  system. 

Of  course,  other  important  process  improvements  have  taken  place  since  the  organization 
achieved  CMMI  maturity  level  5.  Two  that  are  particularly  important  for  the  performance 
results  presented  in  this  case  description  are  improvements  to  the  organization’s  Web-based 
defect  tracking  system  and  its  earned  value  tracking  system.  The  defect  tracking  system 
provides  support  for  identifying  and  tracking  defects  through  corrective  action.  Its  underlying 
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database  provides  information  for  CAR-related  activities,  which  has  led  to  shorter  learning 
and  improvement  cycles  both  within  and  across  projects.  The  basics  of  earned  value  tracking 
were  in  place  earlier;  however,  the  process  has  matured  substantially  since  the  organization 
reached  CMMI  maturity  level  5.  The  resultant  organizational  baselines  have  been  quite 
valuable  for  the  organization’s  OPP-  and  QPM-related  activities. 

Performance  Results 

This  process  standardization  effort  within  the  Software  Engineering  Division  eliminated  late 
deliveries  of  software  releases  from  August  2004  to  January  2005.  This  marked  improvement 
prompted  the  division  to  embrace  a  goal  of  100  percent  on-time  delivery  for  fiscal  year  2005. 
Similar  dramatic  results  were  also  achieved  in  delivered  defect  reduction  after  implementing 
the  CMMI  maturity  level  5  process  improvement  practices  with  no  defects  reported  in  fielded 
software  during  the  same  six  month  period. 

A  recent  study  confirmed  the  dramatic  improvements  in  both  cost  and  schedule  variances. 

The  study  compared  two  samples  of  completed  projects  that  included  both  hardware  and 
software  releases.  The  samples  were  selected  randomly,  stratified,  and  balanced  to  include 
similar  projects  and  project  managers.  The  first  sample  included  projects  completed  prior  to 
the  CMMI  improvement  initiative,  and  the  second  sample  consisted  of  projects  completed 
during  the  past  six  months  when  the  CMMI  processes  were  institutionalized. 

Figure  48  summarizes  the  differences  in  cost  variance.  The  y-axis  shows  the  number  of 
projects,  while  the  range  of  cost  variance  is  shown  across  the  x-axis.  As  can  be  seen  in  the 
figure,  there  was  substantially  more  variation  among  the  projects  cost  performance,  and  more 
of  the  projects  came  in  over  budget  prior  to  the  organization’s  move  to  CMMI  maturity  level 
5.  After  the  Software  Engineering  Division  achieved  CMMI  maturity  level  5,  there  was  much 
less  variation  among  the  projects.  There  was  very  little  variance  between  estimated  and  actual 
cost,  and  proportionally,  more  of  the  projects  were  completed  under  their  cost  estimates. 

Due  to  the  number  of  extreme  values,  the  median  (middle)  values  of  cost  variance  were  used 
for  the  statistical  comparison.  The  95  percent  confidence  interval  of  the  median  cost  variance 
among  the  projects  that  were  completed  prior  to  the  organization’s  achievement  of  CMMI 
maturity  level  5  ranged  from  -6.4  percent  to  +7.78  percent.  The  same  confidence  interval 
after  achievement  of  maturity  level  5  was  -1.0  percent  to  +7.4  percent.  Negative  cost 
variances  are  undesirable  when  representing  cost  over-runs.  The  median  cost  variance 
improved  from  1.4  percent  to  4.4  percent,  and  negative  occurrences  were  almost  completely 
eliminated.  This  improvement  is  significant  at  the  84  percent  confidence  level  using  the 
Mann- Whitney  U  test  [Sheskin  03]. 
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Figure  48:  Change  in  Percent  Cost  Variance 

As  seen  in  Figure  49,  the  performance  effects  are  even  more  striking  with  respect  to  schedule 
variance.7  Several  products  were  delivered  late,  with  considerable  schedule  variance,  prior  to 
the  achievement  of  CMMI  maturity  level  5.  After  achieving  maturity  level  5,  variation  in 
schedule  adherence  was  significantly  reduced  and  negative  occurrences  were  almost 
completely  eliminated. 

Again,  due  to  the  number  of  extreme  values,  the  median  values  of  schedule  variance  were 
used  for  comparison.  The  95  percent  confidence  interval  of  the  median  schedule  variance  was 
-57  percent  to  +2  percent  among  the  projects  that  were  completed  before  the  organization 
achieved  CMMI  maturity  level  5.  The  same  confidence  interval  after  level  5  was  achieved 
ranged  from  -2.5  percent  to  +0.2  percent.  Similar  to  cost  variances,  these  negative  variances 
are  undesirable  schedule  overruns.  The  median  schedule  variance  improved  from  -15.8 
percent  to  almost  zero.  The  confidence  level  is  91  percent  using  Mann- Whitney  U  test 
criteria. 


7  In  order  to  keep  the  x-axes  comparable  visually,  one  extreme  outlier  in  the  Before  CMMI  ML5 
section  has  been  removed  from  the  graphic;  however,  all  26  cases  are  included  in  the  statistical 
analysis. 
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Figure  49:  Change  in  Percent  Schedule  Variance 

Comparable  work  about  defect  reduction  was  ongoing  at  the  time  that  this  SEI  technical 
report  was  published.  Similar  results  were  expected;  however,  as  previously  noted,  delivered 
defects  already  were  very  rare  given  the  nature  of  the  organization’s  life  critical  products. 
Current  work  in  the  402d  Software  Maintenance  Group  is  exploring  the  performance 
effects  that  can  be  expected  from  further  attention  to  reducing  pre-release  defects. 
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5  Summary  and  Conclusions 


Much  has  been  learned  since  the  SEI  report  over  two  years  ago  on  the  impact  and  benefits  of 
CMMI -based  process  improvement  [Goldenson  03].  At  that  time,  numerous  studies  already 
described  the  value  of  process  improvement  based  on  the  SW-CMM  for  the  development  and 
maintenance  of  software  and  software-intensive  systems  [Benno  95,  Butler  95,  Dion  92,  Dion 
93,  Herbsleb  94,  Humphrey  91,  Lebsanft  96,  Lipke  92,  McGarry  98,  Wohlwend  93].  As  seen 
in  this  technical  report,  similar  performance  results  now  exist  for  CMMI-based  improvement. 

The  results  are  drawn  from  organizations  throughout  the  world.  Their  process  improvement 
efforts  cover  both  small  and  large  organizational  units,  and  they  do  business  in  a  variety  of 
sectors  and  domains.  They  apply  CMMI  model  practices  to  systems  engineering  and  various 
other  engineering  disciplines  in  addition  to  software.  While  most  of  the  results  come  from 
higher  maturity  organizations,  notable  improvement  also  have  been  achieved  by  lower 
maturity  organizations. 

Credible  quantitative  results  exist  for  all  six  performance  categories  discussed  in  the  report: 
cost,  schedule,  productivity,  product  quality,  customer  satisfaction,  and  return  on  investment. 
Moreover,  many  of  the  organizations  described  in  this  report  that  have  achieved 
improvements  in  product  quality  and  customer  satisfaction  also  have  achieved  higher 
productivity,  cost  performance,  and  schedule  performance.  Better  quality  may  not  always  be 
free,  but  it  can  occur  with  better  project  performance  as  a  result  of  disciplined  process 
improvement. 

Organizations  that  base  their  process  improvement  activities  on  CMMI  models  can  and  have 
achieved  marked  performance  improvements,  but  more  remains  to  be  learned.  While  case 
studies  provide  a  great  deal  of  valuable  detail  and  context,  their  results  cannot  necessarily  be 
generalized  elsewhere.  A  better  understanding  of  the  reasons  for  varying  success  also  is 
necessary.  An  equally  important  task  is  to  obtain  more  evidence  about  the  statistical 
relationships  between  process  capability  and  program  performance,  along  with  the 
organizational  and  product  characteristics  that  may  affect  them  both. 

Several  quantitative  studies  of  improvement  based  on  the  SW-CMM  systematically  make 
comparisons  across  many  projects  or  organizations  [Clark  97,  Deephouse  95,  El  Emam  00, 
Goldenson  95,  Goldenson  99,  Harter  00,  Herbsleb  97,  Jung  03,  Krasner  99,  Krishnan  99, 
Lawlis  95].  Those  studies  find  evidence  of  considerable  differences  in  product  quality  and 
efficient  delivery  that  vary  predictably  with  differences  in  process  capability  and 
organizational  maturity.  Work  currently  is  underway  at  the  SEI  and  elsewhere  to  conduct 
similar  studies  that  focus  explicitly  on  CMMI-based  improvement. 
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Appendix  A  CMMI  Performance  Results 

Sources  Listed  by 
Organization 


With  the  exception  of  the  original  materials  provided  for  use  in  this  report,  this  appendix 
includes  the  publicly  available  sources  for  all  of  the  results  that  are  reported  in  this  document. 
Other  references  in  the  document  are  listed  in  the  References  section. 
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Conference  and  User  Group.  Denver,  CO,  November  15,  2004. 
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The  CMMI  Performance  Results  Web  site  can  be  found  at 

http://www.sei.cmu.edu/cmmi/results.html.  Figure  50  contains  a  screenshot  of  the 
Performance  Results  main  page. 

The  SEI  updates  performance  results  periodically  and  may  be  able  to  use  your  evidence  if 
you  are  willing  to  share  it  with  others  publicly  or  under  the  promise  of  non-disclosure. 
Results  from  service,  acquisition,  systems  engineering,  hardware  and  software  development 
organizations  are  welcome. 


Software  Engineering  Institute 


Home  Search  Contact  Us  Site  Map  1/WiarlsAtow 


Improving 

management 

practices 


About  Management  Engineering  Acquisition  Weak  with  Us  Products  Publications 
tii*  SEt  and  Services 


MANAGEMENT 


CMMI  Performance  Results 


O  CMMI  Main 
Page 

o  What  Is  CMMI? 
°  Models 
°  Adoption 

°  Training, 
Events,  & 
Forums 

•  Performance 
Results 

°  Appraisals 

°  Frequently 
Asked 


Objective  and  Scope  |  Results  |  Providing  Results 


Objective  and  Scope 


There  is  a  widespread  demand  for  credible,  quantitative 
evidence  about  the  results  of  process  improvement  based 
on  CMMI  models.  The  results  presented  here  are  from 
publicly  available  conference  presentations,  published 
papers,  and  individual  collaborations  with  the  SEI. 


CMMI 


Together,  these  results  provide  proof  of  concept  about  the  potential  of 
CMMI-based  process  improvement.  The  results  show  that  CMMI  often  leads  to 
very  impressive  improvements  in  product  quality,  project  performance,  and 
organizational  performance;  however,  the  individual  results  presented  here  may 
not  be  repeatable  in  every  organization. 


Figure  50:  Screenshot  of  Performance  Results  Main  Page 

Brief  statements  and  graphical  examples  of  the  results  can  be  displayed  by  one  of  the  six 
performance  categories  described  in  this  report  (Figure  51)  or  by  organization  (Figure  52). 
Figure  53  shows  a  screenshot  of  the  Assertion  Statement  Detail  page,  and  this  page  can  be 
seen  by  clicking  View  for  any  of  the  assertion  statements.  Full  source  documents  are 
available  for  some  assertion  statements  and  can  be  seen  by  clicking  View  on  the  Assertion 
Statement  Detail  page. 
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Clvllvll '  Performance  Results 


View  by  Performance  Category 

The  performance  results  examples  contain  brief  assertion  statements  and  their 
sources  and  sometimes  are  accompanied  by  graphic  illustrations.  To  view  the 
graphic  or  source  for  a  statement,  click  the  View  link. 

Cost  |  Schedule  |  Productivity  |  Quality  |  Customer  Satisfaction  |  Return  on 
Investment 


Cost 

Assertion  Statement 

Organization 

20  percent  reduction  in  unit  software  costs  as 

Lockheed  Martin 

the  organization  integrated  its  engineering 

Management  and  Data 

processes 

Systems 

15  percent  decrease  in  defect  find  and  fix 

Lockheed  Martin 

costs  as  the  organization  integrated  its 

Management  and  Data 

engineering  processes 

Systems 

Reduced  cost  of  poor  quality  from  over  45 

Siemens  Information 

percent  to  under  30  percent  over  a  three  year 
period  as  the  organization  moved  from 
SW-CMM  maturity  level  5  towards  CMMI 
maturity  level  5 

Systems  Ltd. 

5  percent  improvement  in  cost  performance 

Raytheon  North  Texas 

index  with  a  34  percent  decline  in  variation  as 
the  organization  improved  from  SW-CMM 
maturity  level  4  to  CMMI  maturity  level  5 

Software  Engineering 

Costs  dropped  43  percent  from  a  baseline 
prior  to  SW-CMM  maturity  level  2  as  the 
organization  moved  toward  CMMI  maturity 
level  3 

DB  Systems  GambH 

Figure  51:  View  by  Performance  Category 
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Lockheed  Martin  Management  and  Data  Systems 

The  performance  results  examples  contain  brief  assertion  statements  and  their 
sources  and  sometimes  are  accompanied  by  graphic  illustrations.  To  view  the 
graphic  or  source  for  a  statement,  click  the  View  link. 

Cost  |  Productivity  |  Customer  Satisfaction  | 


Cost 

Assertion  Statement 


20  percent  reduction  in  unit  software  costs  as  the 
organization  integrated  its  engineering  processes 

^"10^  15  percent  decrease  in  defect  find  and  fix  costs  as  the 
organization  integrated  its  engineering  processes 

4.5  percent  decline  in  overhead  rate  as  the  organization 
improved  from  SW-CMM  maturity  level  3  to  CMMI  maturity 
level  5 

top  Jk 


Productivity 

Assertion  Statement 


30  percent  increase  in  software  productivity  as  the 
organization  integrated  its  engineering  processes 

top  A 


Customer  Satisfaction 

Assertion  Statement 


Increased  award  fees  by  55  percent  compared  to  an  earlier 
SW-CMM  baseline  at  maturity  level  2 

top  A, 


Figure  52:  View  by  Organization 
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Assertion  Statement  Detail 


Statement 


15  percent  decrease  in  defect  find  and  fix  costs  as  the  organization  integrated  its  engineering 
processes 


Organization 


Lockheed  Martin  Management  and  Data  Systems 


Graphic 


Source 


Lockheed  Martin  Integrated  Systems  and  Solutions.  "Key  Business  Indicator  Trends  During  the 
Journey  from  SW-CMM  Level  2  to  CMMI  Level  5  at  Lockheed  Martin  Management  &  Data 
Systems."  McLoone,  Peter.  CMMI  Technology  Conference.  Denver,  CO,  November  2003. 


Figure  53:  View  Detail 
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